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ABSTRACT 

This report presents revised results from the first cycle of 
the Program for International Student Assessment (PISA). PISA is a new 
international assessment of 15-year-olds in reading, mathematics, and science 
literacy. The United States has joined with 27 other member countries of the 
Organization for Economic Cooperation and Development (OECD) and 4 non-OECD 
countries to assess their students' performance against international 
benchmarks. PISA content is not drawn strictly from school curricula, but . 
rather form a framework agreed to nationally in what reading, mathematics, 
and science literacy mean. After the year 2000, PISA results are scheduled to 
be available every 3 years so that progress can be tracked over time. In PISA 
2000, student questionnaires sought information about student attitudes 
toward reading and learning strategies. This information and achievement data 
provide baseline information for later PISA studies. Four appendixes contain 
technical notes, descriptions of international studies in subject areas, 
supporting statistical data, and the released items from PISA 2000 
assessments. (Contains 38 tables, 26 figures, and 41 references.) ( SLD) 
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Commissioner’s Statement 

The Program for International Student Assessment is a new international assessment of 15- 
year-olds in reading, mathematics, and science literacy. We know that assessments serve many 
different purposes. At the classroom level, assessments help teachers determine how 
individual students are doing and what topics may need additional instruction. At the state 
and national level, they help administrators understand overall patterns of student 
performance within states and across the nation. Similarly, at the international level, 
assessments like PISA help policymakers, researchers, and the public understand how the 
performance of their students compares to that of peers in other countries. 

NCES has long participated in both national and international assessments in order to fulfill 
its mission of reporting on the “condition of education in the United States and other 
nations.” PISA adds another facet to NCES’ collection of information. The United States 
joined with 27 other member countries of the Organization for Economic Cooperation and 
Development (OECD) and 4 non-OECD countries to assess their students’ performance 
against international benchmarks through PISA. This OECD-sponsored assessment will 
provide for a regular cycle of information to compare trends over time and across nations. 

PISA is built on a different framework from other national and international studies. It aims 
to advance understanding of how well equipped young people are for their future lives, by 
emphasizing items that have a real-world context. PISA content is not drawn strictly from 
school curricula, but rather from a framework agreed to internationally on what reading, 
mathematics, and science literacy mean. 

Consider the possible experiences of 15-year-olds who took the PISA assessment in the United 
States in the year 2000. Most of them will have been in school for more than 10 years. Some 
will have attended public school, some private school. Some will have come from literacy rich 
home environments. Some will have taken advanced courses, some will have gone on field 
trips, and some will have used computers in their classrooms. In addition to school, some will 
have attended camp, had summer or part-time jobs, cared for younger siblings, or participated 
in sports teams, competitions, or any number of other kinds of activities. All of these 
experiences may be associated with their performance on PISA. The title of this report, 
Outcomes of Learning, reflects PISA’s emphasis on describing performance based on the 
school and non-school learning experiences 15-year-olds may have had. 

The results presented in this report represent only the first cycle of PISA. After 2000, 

PISA’s results are scheduled to be available every 3 years, so that progress for each of the 
three subjects can be tracked over time. Future cycles of PISA will also further address new 
areas of assessment, such as problem solving and the use of information and 
communications technologies. 

We here at NCES hope that the information in this report, and what will be available from 
future phases of PISA, will be of use to all readers — those from the education community, the 
business community, and the general public — who have an interest in understanding the 
performance of America’s 15-year-olds in reading, mathematics, and science literacy. 





Gary W. Phillips 
Deputy Commissioner 
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PISfl in Brief 

The Program for International Student 
Assessment (PISA) is a new system of 
international assessments that focus on 15-year- 
olds’ capabilities in reading literacy, mathematics 
literacy, and science literacy. PISA also measures 
general or cross-curricular competencies such as 
learning strategies. 

PISA will be implemented on a 3-year cycle that 
began in 2000. Each PISA assessment cycle 
focuses on one particular subject, although all 
three are assessed in each cycle. In this first cycle, 
PISA 2000, reading literacy is the major focus, 
occupying roughly two-thirds of assessment time. 
In 2003, PISA will focus on mathematics literacy, 
and in 2006, on science literacy (figure 1). 

PISA will report on performance in reading 
literacy, mathematics literacy, and science literacy 
every 3 years, and provide a more detailed look 
at each domain in the years when it is the major 



focus. For instance, this report will provide 
average scores for specific reading processes such 
as retrieving information, interpreting texts, and 
reflecting on texts, as well as a combined reading 
literacy average score. Only single measures of 
mathematics and science literacy are available in 
PISA 2000, with more specific information to be 
provided for these domains in subsequent cycles. 
These cycles will allow countries to compare 
changes in trends for each of the three content 
areas over time. Future cycles will also include 
further development of the assessment of cross- 
curricular competencies, such as problem solving 
in 2003 and use of information and 
communications technology in 2006. 

PISA is sponsored by the Organization for 
Economic Cooperation and Development 
(OECD), an intergovernmental organization of 
30 industrialized nations that serves as a forum 
for member countries to cooperate in research 
and policy development on social and economic 
topics of common interest. PISA is a 
collaborative venture, with representatives from 



Figure 1 . — Program for International Student Assessment (PISA) assessment cycle 







y 



NOTE: The subject in all capital letters in each assessment cycle is the major domain for that cycle. 

SOURCE: Organization for Economic Cooperation and Development, Program for International Student Assessment (PISA) 2000. 
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Figure 2. — Participating countries in the Program for International Student Assessment (PISA) 2000 




OECD countries 



Australia | 

Austria 

Bel g ium 1 

Canada 

Czech Republic | 

Denmark 

Finland | 

France 

Germany | 

Greece 

Hun gar y | 

Ice l and 

Ireland | 

Italy 



Japan ] 

Korea, Republic of 

Luxembour g ] 

Mexico 

Netherlands | 

New Zealand 

Norway [ 

Poland 

Portu g al | 

S pain 

Sweden | 

Switzerland 

United Kin g dom | 
United States 



Non-OECD countries 



Brazil [ [ Liechtenstein 

Latvia Russian Federation 




NOTE: Although the Netherlands participated in the Program for International Student Assessment (PISA) in 2000, technical problems 
with its sample prevent its results from being discussed in this report. For information on the results for the Netherlands, see OECD 
( 2001 ). 

SOURCE: Organization for Economic Cooperation and Development, Program for International Student Assessment (PISA) 2000. 



member country governments jointly steering the 
project through a Board of Participating 
Countries. At the international level, the 
Australian Council for Educational Research 
(ACER) leads a consortium that coordinates PISA 
under direction from the OECD. 1 In the United 
States, the National Center for Education 
Statistics (NCES) is responsible for U.S. data 
collection and represents the United States in the 
international management of the assessment. 
Westat, a private research firm, handled data 



collection in the United States for PISA 2000 
under contract to NCES. 

In 2000, 32 countries participated in PISA, 
including 28 OECD countries and 4 non-OECD 
countries (figure 2). 2 

To implement PISA 2000, each participating 
country selected a nationally representative 
sample of 15-year-olds. In the United States, this 
included nearly 4,000 students from both public 



1 Other members of the PISA Consortium include the Netherlands National Institute for Educational Measurement (CITO), Educational 
Testing Service (ETS, USA), the National Institute for Educational Research (NIER, Japan), and Westat (USA). 



2 Another 12 countries will carry out a second round of the PISA 2000 assessment in 2002. 
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and nonpublic schools (table Al.l) from several 
different grade levels. 3 Appendix 1, Technical 
Notes, contains more information about sampling 
and other aspects of PISA 2000’s design. Each 
selected student completed an approximately 90- 
minute assessment and a 20- to 30-minute 
questionnaire designed to gather information 
about his or her background and experiences 
related to reading, mathematics, and science 
literacy. Principals in schools where students took 
the PISA assessment also completed a 
background questionnaire about their schools. 
PISA 2000 consisted of a mix of multiple choice, 
short answer, and extended response questions. 
Assessments were conducted in the United States 
in the spring of 2000 by trained test 
administration field staff that visited each of the 
participating schools and administered both the 
assessments and the questionnaires. 

PISA’s Yield Measure of Learning 

PISA’s purpose is to represent the overall yield of 
learning for 15-year-olds. This yield is the sum of 
learning outcomes for 15-year-olds in reading, 
mathematics, and science literacy and is 
represented by national averages of student 
scores. PISA assesses the cumulative educational 
experiences of all students who are 15 years of 
age at the time of assessment, irrespective of the 
grade levels or type of institutions in which they 
are enrolled. PISA assumes that by the age of 15, 
young people have had a series of learning 
experiences, both in and out of school, that allow 
them to perform at particular levels in reading, 
mathematics, and science literacy. Clearly, formal 
education will have played a major role in their 
performance, but other factors, such as learning 
opportunities at home or elsewhere outside of 
school, also play a role. PISA’s results provide a 
valuable indicator of the overall performance of a 
country’s education system, but they also provide 



information about other factors that influence 
performance. 

By assessing students near the end of compulsory 
schooling in key knowledge and skills, PISA 
provides information about how well prepared 
students will be for their future lives as they 
approach an important transition point for 
education and work. PISA is forward rather than 
backward looking in this sense, since it aims to 
show how equipped 15-year-olds are for their 
futures based on what they have learned up to 
that point. 

The Unique Contribution of PISA 

PISA grew out of OECD efforts to develop 
comparable measures of learning outcomes for 
policy use. By creating PISA, OECD member 
countries sought to develop a regular cycle of 
data collection in key areas. These data will 
provide information at the national and 
international level about how well countries are 
meeting their educational objectives. The OECD 
will use the data to produce indicators of 
educational systems, which provide a 
“quantitative description of the functioning of 
education systems that allows countries to see 
themselves in the light of other countries’ 
performance” (OECD 1998, p.5). 

A number of international comparative studies 
already exist to measure achievement in 
mathematics, science, and reading, including the 
Trends in International Mathematics and Science 
Study (TIMSS) and the Progress in International 
Reading Literacy Study (PIRLS). The Adult 
Literacy and Lifeskills survey (ALL) will measure 
the reading literacy skills of adults. In addition, 
the United States has been conducting its own 
national surveys of student achievement for more 
than 30 years through the National Assessment 
of Educational Progress (NAEP) program. 



3 



For information on distributions of students by grade in participating countries, see table A1.2, appendix 1. 
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Appendix 2 gives an overview of international 
studies in reading, mathematics, and science 
(tables A2.1, A2.2, and A. 2. 3). PISA differs from 
these studies in several ways. 

o Content . While other studies, such as 
TIMSS and NAEP, have a strong link to 
curriculum frameworks and seek to measure 
students’ mastery of specific knowledge, 
skills, and concepts, PISA is designed to 
measure “literacy” more broadly. PISA’s 
content is drawn from broad content areas, 
such as space and shape for mathematics, in 
contrast to more specific curriculum-based 
content such as geometry or algebra. 

© Tasks . In addition to the differences in 
purpose and age coverage between PISA and 
other international comparative studies, 
what students are asked to do on PISA tasks 
is also somewhat different. A study based 
on expert panels’ reviews of mathematics 
and science items from PISA, TIMSS, and 
NAEP reports that PISA items require 
multistep reasoning more often than either 
TIMSS or NAEP (Nohara 2001). The study 
also shows that both PISA mathematics and 
science literacy items often involve the 
interpretation of charts and graphs or other 
“real world” material. The unique 
contribution of PISA lies in its focus on 
assessing students’ knowledge and skills in 
reading, mathematics, and science in the 
context of everyday situations. These tasks 
reflect the underlying assumption of PISA: 
as 15-year-olds begin to make the transition 
to adult life, they need to know not only 
how to read, or understand particular 
mathematical formulas or scientific concepts, 
but also how to apply this knowledge and 
these skills in the many different situations 
they will encounter in their lives. 

© Age-Based Sample . PISA collects 
information from an age-based sample, 



rather than a grade-based sample. Schools 
identify students who are 15 years of age, 
regardless of what grade they are in. In 
contrast, PIRLS, for example, collects 
reading literacy data for fourth-grade 
students, TIMSS 1999 collected 
mathematics and science data for eighth- 
grade students, and NAEP (main) collects 
data at various grade levels. 4 PISA uses an 
age-based sample for several reasons. First, 
PISA’s goal is to represent outcomes of 
learning rather than outcomes of schooling. 
By placing the emphasis on age, PISA 
intends to show not only what 15-year-olds 
have learned in school, but outside of school 
as well and over the years, not just in a 
particular grade. In addition, years of 
education vary among countries (for 
example, 10th grade in the United States 
may not correspond to a similar educational 
level in other countries). Choosing an 
age-based sample makes comparisons across 
countries somewhat easier. One other 
international study does collect an age-based 
sample. The Adult Literacy and Lifeskills 
survey (ALL) will collect reading literacy 
data for adults aged 16 to 65. Although the 
ALL measures of reading literacy are slightly 
different than those for PISA, there will be 
efforts to link the performance of 15-year- 
olds to that of adults through a common set 
of items in order to examine the relationship 
of literacy to the labor market and other 
facets of adult life. 

<r> Age Level . Since PISA seeks to show the 
overall yield of an educational system and the 
cumulative effects of all learning experiences, 
the age of 15 was chosen to represent a point 
in time at which these broad learning 
outcomes could be measured while all 
students were still required to be in school. 
The grade levels covered in other 
international assessments correspond roughly 
to the ages of 9, 13, and 17. 



4 TIMSS does use some age criteria, but the sample of students is drawn from one grade rather than across grades as in PISA. For 
example, TIMSS 1999 required participating countries to assess students in the upper of the two grades with the largest proportion of 
13 -year-o Ids. 
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o Information Collected . The kind of 
information PISA collects also reflects its 
broad policy purpose. For example, in 
contrast to PISA, TIMSS collects background 
information intended to help provide an 
understanding of how teachers in different 
countries approach the task of teaching and 
provide insight into what effects these 
different approaches might have on student 
performance. The TIMSS video studies 
extend this work even further by actually 
capturing images of instruction across 
countries. PISA, on the other hand, collects 
only background information related to 
general school context and student 
demographics. No teacher questionnaires are 
included in this cycle of PISA. While its 
results can certainly inform education policy 
and spur further investigation into differences 
within and between countries, PISA is not 
meant to provide direct information about 
improving education in the classroom. Its 
purpose is to generate useful indicators to 
benchmark performance and inform policy. 

The United States has been actively involved in 
the development of PISA since its inception, 
believing that PlSA’s differences from other 
studies allow it to complement the picture of U.S. 
performance obtained from other studies and 
provide a new perspective on U.S. education in an 
international context. 



Report Summary and 
Organization 

This report focuses on U.S. results for PISA 2000. 
The OECD is also releasing a report discussing 
PISA 2000 results, but from an international 
perspective. The OECD report is being released 
at the same time as this U.S. national report, 
providing a wealth of information on PISA 2000. 

The following chapters describe in detail PlSA’s 
definitions for reading literacy, mathematics 
literacy, and science literacy and U.S. performance 
on each of these measures. Chapter two discusses 
reading literacy and chapter three describes 
mathematics and science literacy. In addition to a 
discussion of national averages, chapters two and 
three take a closer look at the distributions of 
literacy across countries, including percentages of 
15-year-olds meeting international benchmarks. 
Each of these chapters also describes sample PISA 
2000 items, and discusses U.S. and international 
performance on selected items. 

Chapter four describes differences in 
performance as they relate to demographic 
characteristics such as gender, race and ethnicity, 
parents’ education, and others. Chapter five 
briefly discusses some examples of the general or 
cross-curricular competencies that will take on a 
growing role in PISA in future cycles, including 
attitudes toward learning and learning strategies 
such as memorization and elaboration. Finally, 
appendices provide technical information on 
how PISA 2000 was conducted, supporting 
statistical detail for the figures in the text, an 
overview of how PISA compares to other 
international studies, and a set of released sample 
PISA 2000 items for reading, mathematics, and 
science literacy. 
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Key Findings 

o On the combined reading literacy scale for PISA 2000, U.S. 15-year-olds 
perform about as well on average as 15-year-olds in most of the 27 
participating OECD countries. Students in Canada, Finland, and New Zealand 
outperform U.S. students. U.S. students perform at the same level as students in 
19 other participating OECD countries and Liechtenstein. U.S. students 
perform better on average than students from the OECD nations of Greece, 
Luxembourg, Mexico, and Portugal (figure 3; table A3.1). 

o For each of the three specific reading process subscales, retrieving information , 
interpreting texts , and reflecting on texts, U.S. scores are not different from 
the OECD averages. Canada and Finland outscore the United States on each of 
the three reading process subscales, and the United States outscores at least 
seven other nations on each measure (figure 3; table A3.1). 

o Fifteen countries, or about half of the countries participating in PISA 2000, 
show less variation in student performance than the United States. The 
remaining countries show similar variation in student performance to the 
United States, and U.S. variation is similar to the OECD average (table A3.3). 



o The top 10 percent of OECD students score 623 or higher on the combined 
reading literacy scale. In the United States, 13 percent of students achieve this 
score or better, a percentage not different from the OECD top 10 percent 
benchmark. Three countries (Canada, Finland, and New Zealand) have a 
higher percentage of students score in the top 10 percent, while 14 countries 
have a lower percentage (figure 5; table A3.4). 

o Percentages of U.S. students across the literacy levels are similar to the OECD 
average percentages, except at level 5. In the United States, 12 percent of 15- 
year-olds read at level 5, the highest proficiency level, a percentage higher than 
the OECD average. Level 1 encompasses 12 percent of students, and 6 percent 
of U.S. 15-year-olds are below level 1 (figure 8; table A3. 5). 



BEST COPY AVAILABLE 



o Looking at the cumulative percentages of students from level to level, about 
one-third of U.S. students perform at the two highest levels, level 4 and level 
5. In Finland, about half of the students perform at levels 4 and 5, and in 
Brazil, 4 percent of students do. About 60 percent of students in the United 
States perform at level 3 or above, and over 80 percent at level 2 or above. 
Finland, with the highest average combined reading literacy score, has 79 
percent of students at level 3 or above, and 93 percent of students at level 2 or 
above (table A3. 5). 
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Reading literacy is key in an information- 
abundant world. PISA builds upon the work of 
previous U.S. national and international studies in 
defining and reporting on reading literacy. This 
chapter describes the definition and reporting 
scales for reading literacy in PISA 2000, in which 
reading literacy is the major subject covered, and 
provides information on U.S. performance in an 
international context. Beginning with a 
description of national average scores, the 
discussion then turns to distributions of high- 
performing students, then to overall distributions 
of student scores, and finally to groups of 
students with particular sets of skills. 



Defining Reading Literacy 

PISA defines reading literacy as: 



Understanding , using, and reflecting on 
written texts in order to achieve one's 
goals, to develop one's knowledge and 
potential, and to participate in society 
(OECD 1999, p.20). 



Since PISA measures the achievement of 15-year- 
olds, it does not focus on the most basic reading 
skills. Instead, PISA seeks to measure the extent 
to which students can “construct, extend, and 
reflect on the meaning of what they have read” 
across a wide variety of texts associated with a 
wide variety of situations. PISA reading literacy 
tasks were constructed within three main 
dimensions. Mathematics and science literacy 
items fall within similar dimensions, as will be 
seen in the following chapter. The dimensions for 
the reading literacy tasks are: 

® Content or Structure — refers to types of 
text, such as continuous and noncontinuous 
texts. Continuous texts are prose texts that 
are largely composed of sentences organized 
into paragraphs. Noncontinuous texts are 
those that are often organized as lists or 
charts (sometimes referred to as documents). 



<s> Processes — consists of the kinds of processes 
used when reading a text, including 
retrieving information, understanding texts 
at a general level, interpreting texts, 
reflecting on content of texts, and reflecting 
on form of texts. 

® Situations — distinguishes the use for which 
texts were constructed or the context in 
. which knowledge and skills are applied, such 
as private use, public use, occupational use, 
or educational use. For example, private use 
refers to novels or personal letters, public use 
refers to official documents or 
announcements, occupational use refers to 
manuals or reports, and educational use 
refers to textbooks or worksheets. 

In short, PISA measures how well 15-year-olds 
are able to apply different reading processes to a 
wide range of reading materials, such as the kinds 
of forms they receive from their governments, the 
kinds of articles they read in their local 
newspapers, the kinds of manuals they read for 
work or school, or the kinds of books or 
magazines they read for entertainment. 

The basic form of the assessment reflects this 
range of materials and processes. Each reading 
literacy assessment unit consists of a passage of 
text, followed by a number of questions, some 
with a multiple-choice format and others 
requiring students to construct their own 
answers. Examples of reading assessment items 
are described later in this chapter and can be 
found in appendix 4. 



Specific Skill Areas 
in Reading Literacy 

PISA 2000 provides information on three specific 
reading skill areas derived from the processes 
described above for gaining meaning from a text, 
retrieving information, interpreting texts, and 
reflecting on texts. 

Retrieving information — the ability to locate 
one or more pieces of information in a text. All 
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tasks require locating information in the text. 

The difficulty of any task is determined by how 
much information needs to be accessed, how 
explicitly it is signaled in the text, and whether 
the required pieces of information are 
interrelated or independent. 

Interpreting texts — the ability to construct 
meaning and draw inferences from one or more 
parts of a text. The easiest tasks require 
identifying a main idea in the text. More difficult 
tasks require understanding relationships within 
the text that are an inherent part of its 
organization and meaning — that is, 
understanding how language is being used to 
convey meaning in context and comparing, 
contrasting, and/or categorizing ideas. 

Reflecting on texts — the ability to relate a text 
to one’s own experience, knowledge, and ideas. 
The easiest tasks require making a basic 
connection between the text and what the reader 
already knows. More difficult tasks involve 
comparisons between and/or a synthesis of 
information from the two sources. 

Specific information on reading literacy 
proficiency could also be derived from the 
contents/structures or situations described above, 
as has been done in previous large-scale studies, 
including the International Association for the 
Evaluation of Educational Achievement (IEA) 
Reading Literacy Study (which describes literacy 
performance for narratives, exposition, and 
documents) and in the International Adult 
Literacy Study (which describes literacy 
performance for prose and documents). 

However, the emphasis on reading processes 
reflects the policy objectives of PISA most closely, 
and it is hoped that the development of three 
reading literacy process scales for PISA 2000 will 
provide a unique insight into the understanding 
of reading literacy. The three process subscales 



are based on the set of five processes described 
above (retrieving information, understanding 
texts at a general level, interpreting texts, 
reflecting on content of texts, and reflecting on 
form of texts). Understanding texts at a general 
level and interpreting texts are grouped together 
because both require a reader to process 
information from either the whole text or one 
part of the text.' Reflecting on content of texts 
and reflecting on form of texts are grouped into 
a single scale because the distinction between 
reflecting on form and reflecting on content, in 
practice, was found to be difficult to make. In 
addition, the amount of information available 
made reporting on three reading literacy 
subscales more feasible than five. 

It should be noted that there is overlap between 
the three subscales that are presented here: in 
practice, most tasks make a number .of different 
demands upon readers, and individual readers 
may approach a task in different ways. Despite 
the interdependence of the three subscales, they 
may reveal interesting and useful distinctions 
both between countries and within countries. 5 
Average scores are reported for each of these 
three reading process subscales. Together, these 
three subscale scores make up the combined 
reading literacy score. 

Reading Literacy in 
PISfl Countries 

Perhaps the simplest and most concise way to 
look at a country’s yield in reading literacy is to 
examine its national average score. Performance 
for 15-year-olds on PISA 2000 is reported as a 
score ranging from 0 to 1,000, with most scores 
falling between 200 and 800. The scale is 
constructed so that the average score for students 
from all OECD countries is 500. 6 Because of the 
statistical techniques used to sample students, 



5 The discussion in this section draws directly from Organization for Economic Cooperation and Development. (2001, April 18-20). 
Described Proficiency Scales for PISA 2000 . Discussion document presented at the PISA Board of Participating Countries meeting, 
Paris, France. 

6 The average score for student performance on each scale (combined reading literacy scale, mathematics literacy scale, science literacy) is 
set at 500 and the standard deviation at 100, with the scale calibrated with equal weightings to results in each countrv. Since the 
calibration was performed on the combined reading literacy scale, average scores for the three reading process subscales differ slightly 
from 500, as shown in figure 3. 
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simply ranking countries based on their average 
score is not accurate. 7 In figure 3, the shading 
identifies countries whose averages are higher, 
lower, or not different from that of the United 
States on the combined reading literacy scale, 
and for each of the three reading process 
subscales. 8 Non-OECD countries are shown at 
the bottom of the figure with shading to indicate 
differences from the United States; however, non- 
OECD countries are not included in determining 
the OECD average. 

On the combined reading literacy scale, U.S. 15- 
year-olds perform about as well on average as 15- 
year-olds in most OECD countries. U.S. students 
perform better than students in the OECD 
countries Greece, Luxembourg, Mexico, and 
Portugal, and the non-OECD nations Brazil, 
Latvia, and the Russian Federation. Students in 
Canada, Finland, and New Zealand outperform 
U.S. students. U.S. students perform at about the 
same level as the other 19 participating OECD 
countries and Liechtenstein (figure 3; table A3.1). 
This finding is generally consistent with previous 
findings of the reading capabilities of U.S. students 
from a 1991 international study of reading 
literacy that placed U.S. 14-year-olds at levels 
similar to other OECD nations, and in which 
Finland outscored the United States (Elley 1992). 

In each of the three reading process subscales, 

U.S. scores are not different from the OECD 
average. Canada and Finland outscore the United 
States on each of the three reading subscales, and 
the United States outscores at least seven other 
nations on each subscale (figure 3; table A3.1). 

More countries outperform the United States in 
retrieving information (five countries) than in 
interpreting texts (two countries) or reflecting on 
texts (four countries). Australia and Korea, for 



instance, perform better than the United States in 
retrieving information , but not differently for 
interpreting texts or reflecting on texts. 

There are clear consistencies across the three 
reading process subscales of retrieving 
information , interpreting texts, and reflecting on 
texts, which carry through to the combined 
reading literacy score. Nations with high scores 
in one area tend to have high scores in the others, 
and the correlations between the combined 
reading literacy scale and the specific reading 
subscales are high (table A3. 2). 



The Distribution of 
Reading Literacy 

National Percentiles 

The average scores for reading literacy describe 
how a country performs overall compared to 
other nations, but they provide no information 
about the way scores are distributed in countries. 
One country with an average score similar to 
another could have large numbers of high- and 
low-scoring students, while the other country 
could have large numbers of students performing 
at about the average score. This section will 
discuss how distributions of scores for the 
combined reading literacy scale compare to one 
another, in order to begin to understand the 
variability of performance in a country as well as 
its average performance. 

Comparing the U.S. average score to 
corresponding cut points in other countries 
provides a means to examine the variation in 
scoring. This can be seen graphically in figure 4, 
in which the 25th percentile in Finland and the 
75th percentile score in Mexico correspond to 
approximately the U.S. average score. This 



7 Average scores for each country are based on a sample of students, rather than all students, and are estimates of the population value of 
all 15-year-olds in each country. These estimates have a known degree of sampling erroi; the standard erroi; and an unknown degree of 
nonsampling error. The true average for any country lies within a range of approximately two times the standard error above and 
below the estimated score. See tables in appendix 3 for standard errors. 

8 Tpiroughout this report, differences between averages or percentages that are statistically significant are described as “higher” or 
“lower.” Differences that are not statistically significant are referred to as “similar to” or “not different from” each other. To determine 
whether differences reported are statistically significant, two-tailed t-tests at the .05 level were used. Bonferroni adjustments were made 
when more than two groups were compared simultaneously. See appendix 1 for more information on statistical procedures used for 
this report. 
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Figure 3. — Combined reading literacy average scores and average subscale scores of 1 5-year-olds, by country: 2000 
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means that 75 percent of Finnish students 
perform above the U.S. average, but 25 percent of 
Mexican students do. 

Another way of looking at variability is to 
consider the number of score points (the size of 
the difference) between groups of students within 
a country. In the United States, for example, the 
5th percentile score for combined reading literacy 
is 320. Ninety-five percent of U.S. students score 
above 320; in the same way, 5 percent of U.S. 
students score above 669, the 95th percentile 
score. This means the top 5 percent of U.S. 
students score at least 350 points higher than the 
bottom 5 percent (table A3. 3). 

Looking at the length of the bars in figure 4 gives 
a sense of how large the differences are between a 
country’s highest and lowest performing students, 
but it does not describe how many students are 
high or low performing. As with average scores, 
because of the statistical techniques used to sample 
students, it is not accurate to rank countries’ 
scoring variation based simply on the length of the 
bars shown in figure 4. Standard deviations of the 
combined reading literacy average scores give a 
mathematical way to tell how greatly scores are 
spread out from the country’s average score (data 
not shown, see table A.3.3). Fifteen countries, or 
about half of the countries participating in PISA 
2000, show less variation in student performance 
than the United States. Fifteen countries show 
similar variation to the United States, and no 
country has greater variation. Some countries that 
perform better on average than the United States, 
such as Canada and Finland, show less variation. 
In contrast, some countries that perform better on 
average also show similar variation, such as New 
Zealand. 

International Percentiles 

Another way to analyze how performance is 
distributed in countries is to look at what 
proportion of students in each country meets 
international benchmarks, standards of 
performance that are applied across countries. 
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The international benchmarks or standards of 
performance used in this case are the percentages 
of students from each country who score in the 
top 10 percent, top 25 percent, top 50 percent, or 
top 75 percent internationally (figure 5; table 
A3.4). 9 Examining the top 10 and top 25 percent 
benchmarks allows a comparison of proportions 
of high-performing students between countries. 

The top 10 percent of OECD students score 623 
or higher on the combined reading literacy scale. 
In the United States, 1 3 percent of students 
achieve this score or better, a percentage not 
different from the OECD top 1 0 percent 
benchmark. Three countries (Canada, Finland, 
New Zealand) have a higher percentage of 
students score in the top 10 percent, while 14 
countries have a lower percentage. Ten of these 
14 countries have 5 percent or less of their 15- 
year-olds score in the top 10 percent. 

The top 25 percent of all students score 571 or 
better on the combined reading literacy scale. 
Twenty-seven percent of U.S. 15-year-olds meet 
this benchmark (figure 5, page 14). Four 
countries have higher percentages of 15-year-olds 
at this benchmark, and 10 have lower 
percentages. Canada, Finland and New Zealand 
again have higher percentages of students meeting 
this benchmark than the United States; in this 
case, Australia’s percentage of students is also 
higher than the U.S. percentage. Again, the U.S. 
percentage is similar to the OECD average, 
suggesting that the United States has proportions 
of high-performing 15-year-olds similar to other 
OECD countries on average. 

This section describes how proportions of U.S. 
high-performing students compare to proportions 
of similarly high-performing students from other 
countries. Results show that the United States 
has similar proportions of students performing at 
each international benchmark to the OECD 
average. The following section takes a more 
detailed approach to analyzing variation between 
countries in student performance, by dividing 



9 To identify the score that separates the top 10 percent of students from the rest, the achievement results of 15-year-olds from all 
participating OECD countries are pooled. Differences in sample size between countries are adjusted so that all nations contribute 
equally to this pool. The 90th percentile of this distribution of scores is the cut point that identifies the top 10 percent benchmark. 
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Figure 4. — Distribution of combined reading literacy scores of 15-year-olds by percentiles, by country: 2000 
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Czech Republic 
Italy 
Germany 
Hungary 
Poland 
Greece 
Portugal 
Luxembourg 
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NOTE: Although the Netherlands participated in the Program for International Student Assessment (PISA) in 2000, technical problems 
with its sample prevent its results from being discussed here. For information on the results for the Netherlands, see OECD (2001). The 
OECD average is the average of the national averages of 27 OECD countries. Because PISA is principally an OECD study, the results for 
non-OECD countries are displayed separately from those of the OECD countries and not included in the OECD average. 

SOURCE: Organization for Economic Cooperation and Development, Program for International Student Assessment (PISA) 2000. 
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Figure 5. — Percentage of 1 5-year-olds reaching the PISA top 10 percent and top 25 percent on the 
combined reading literacy scale, by country: 2000 
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New Zealand 
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Portugal 
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Liechtenstein 
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Russian Federation 
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Russian Federation 
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1 | Percentage is significantly higher than the U.S. percentage 

I | Percentage is not significantly different from the U.S. percentage 



Percentage is significantly lower than the U.S. percentage 



*The shading of Canada in the top 10 percent category may appear incorrect; however, statistically, its placement is correct. 

NOTE: Although the Netherlands participated in the Program for International Student Assessment (PISA) in 2000, technical problems 
with its sample prevent its results from being discussed here. For information on the results for the Netherlands, see OECD (2001). 

SOURCE: Organization for Economic Cooperation and Development, Program for International Student Assessment (PISA) 2000. 
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students into groups based on their performance 
on particular kinds of items. 

Levels of Proficiency 
in Reading Literacy 

Another way to describe performance in reading 
literacy is to examine the proportions of students 
who can accomplish tasks at particular levels. 
This kind of analysis allows a further breakdown 
of average scores and an examination of groups 
of students who show similar skills. In PISA, 
reading literacy is a continuum rather than a 
dichotomy — that is, reading literacy is not 
something you have or don’t have, but rather 
every 15-year-old shows a certain level of literacy 
skills. PISA measures what students can do at 
each of five designated levels. This section 
provides information about PISA 2000 reading 
items and the percentages of U.S. 15-year-olds 
who perform at each level on PISA 2000 in 
comparison to their international peers. 

In order to reach a particular level, a student 
must be able to answer correctly a majority of 



items at that level. 10 This implies that students 
can also correctly answer items below their 
identified level. Students were classified into 
reading levels according to their scores (figure 6). 

A small number of students in each country have 
scores below the lowest of the defined levels, level 
1; that is, they are not able to routinely 
demonstrate the most basic type of knowledge 
and skills that PISA seeks to measure. These 
students score below 335 points on the PISA 
2000 scale. These students may have serious 
difficulties in reading or other learning problems, 
diverse language backgrounds, or they may be 
students who for some other reason cannot or do 
not successfully complete the minimum PISA 
2000 items. These students are not included in 
the proportions for students at level 1, but are 
considered as below level 1 because PISA 2000’s 
descriptions of levels cannot accurately predict 
what skills these students may have. 

PISA 2000 defines five skill levels for the three 
reading processes ( retrieving information , 



Figure 6. — Cut point scores for combined reading literacy levels, by level: 2000 




Level 


Score* 


Below 1 


334 and below 


1 


335 - 407 


2 


408 - 480 


3 


481 - 552 


4 


553 - 625 


5 


626 and above 




* Exact cut point scores are as follows: below level 1: a score equal to or less than 334.75; level 1: a score greater than 334.75 and equal 
to or below 407.47; level 2: a score greater than 407.47 and equal to or below 480.18; level 3: a score greater than 480.18 and equal to 
or below 552.89; level 4: a score greater than 552.89 and equal to or below 625.61; and level 5: a score greater than 625.61. 

NOTE: The Program for International Student Assessment (PISA) uses five levels of performance to describe student performance. In 
order to reach a particular level, a student must be able to correctly answer a majority of items at that level. Students were classified into 
reading levels according to their scores. 

SOURCE: Organization for Economic Cooperation and Development, Program for International Student Assessment (PISA) 2000. 



10 Levels were defined such that students at the top of a level have a 62 percent chance of answering the hardest items in the level 
correctly and students at the bottom of the same level would have a 62 percent chance of answering the easiest items in that level 
correctly. For more information on the process for defining levels, see appendix 1, Technical Notes. 
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interpreting on texts , and reflecting on texts) 
and for a combined reading literacy measure. 

The kinds of tasks that represent each level of 
performance for the specific reading processes are 
described in figure 7. 

Tasks for the combined reading literacy levels are 
defined by using elements from each of the 
specific reading process subscales. For example, 
the lowest level reading literacy tasks require 
students to locate single pieces of information 
with little or no competing information or draw 
simple inferences. The highest-level tasks require 
students to examine very complex texts, locate 
and organize multiple pieces of information, 
interpret language or apply unfamiliar 
categorization schemes, or evaluate and 
hypothesize about the information in the text. 

As the figure illustrates, the level of the tasks for 
the retrieving information scale depends on how 
much information is requested, how clearly it is 
identified in the text, and whether or not the 
information stands alone or is embedded in the 
text. For example, given a short article about 
how good athletic shoes can help prevent injuries, 
students had to locate the answer to the question: 

According to the article, why should sports shoes 
not be too rigid? 

The answer is found at the beginning of a 
paragraph and uses the same wording as the 
question: 

If a shoe is too rigid, it restricts movement. 

To receive full credit, students had to write an 
answer that referred to a restriction of movement, 
either using exactly these words or others that 
convey this idea. This item was considered a level 
1 retrieving information item. A more difficult 
level 4 retrieving information item requires 
students to read an excerpt from a play, and from 
stage directions and dialogue, determine where 
the two characters are located on stage. These 
original passages and others that illustrate both 
the different reading processes ( retrieving 
information , interpreting texts , and reflecting 
on texts) and a variety of literacy levels are 
included in appendix 4. Other sample items may 



be viewed in the OECD initial PISA 2000 report 
Knowledge and Skills for Life— First Results 
from the OECD Programme for International 
Student Assessment or the OECD’s Web Site for 
PISA, www.pisa.oecd.org, by clicking on the 
menu item “sample test items.” 

The easiest tasks in interpreting texts require 
identifying a main idea in a text. For example, a 
level 1 interpreting task using the same article 
about athletic shoes described above requires 
students to identify the author’s intent from a 
multiple choice list. More difficult tasks for 
interpreting texts require understanding 
relationships within a text that are an inherent 
part of its organization and meaning. The most 
difficult tasks are of two kinds. The first requires 
an understanding of how language is being used 
to convey meaning in context, and the second 
requires comparing, contrasting, or categorizing 
ideas in the text. An example of a level 5 
interpreting item is also shown in appendix 4, in 
which students must examine a tree diagram 
describing the labor force structure in a country, 
and then use information from the diagram and 
its footnotes to categorize examples of workers 
into the same structure. 

For reflecting on texts, the easiest tasks require 
making a basic connection between the text and 
the reader’s own knowledge. Again, using the 
athletic shoe article as an example, a level 1 
reflecting item based on the article requires 
students to examine a sentence from the article. 
Using their own knowledge, students have to then 
choose a description of how the parts of the 
sentence relate to one another from a multiple 
choice list, as below: 

Look at this sentence from near the end of the 
article. It is presented here in two parts: 

(first part) “To avoid minor but painful 

conditions such as blisters or even 
splits or athlete’s foot (fungal 
infections),...” 



(second part) “...the shoe must allow 

evaporation of perspiration and 
must prevent outside dampness 
from getting in.” 
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Figure 7. — Reading literacy subscale task descriptions, by level: 2000 




Retrieving information 


Interpreting texts 


Reflecting on texts 


Level 1 


Locate one or more independent 
pieces of explicitly stated 
information, typically meeting a 
single condition or criterion, with 
little or no competing information 
in the text. 


Recognize the main theme or 
author's purpose in a text about a 
fomiliar topic, when the idea is 
prominent or pervasive, either by 
being repeated or by appearing 
early in the text. 


Make a simple connection 
between information in the text 
and common, everyday 
knowledge, with explicit direction 
to consider relevant factors in the 
task and the text. 


Level 2 


Locate one or more pieces of 
information, which may need to 
be inferred, and may need to 
meet several conditions, with 
some competing information 
present in the text. 


Recognize the main idea in a text 
when the information is not 
prominent. Understand 
relationships or construe meaning 
within a limited part of the text, 
making low level inferences. Make 
comparisons or contrasts based 
on only one feature of the text. 


Make a comparison or several 
connections between the text and 
outside knowledge. Draw on 
personal experience and attitudes 
to explain a feature of the text. 


Level 3 


Locate and, in some cases, 
recognize the relationship 
between several pieces of 
information that must meet 
multiple conditions set by the 
question, with prominent 
competing information. 


Integrate several parts of a text in 
order to identify a main idea, 
understand a relationship, or 
construe the meaning of a word or 
phrase. Take into account many 
features in comparing, contrasting 
or categorizing, where required 
information is not prominent. 


Make connections, comparisons, 
and explanations, or evaluate a 
feature of the text. Demonstrate a 
fine understanding of the text in 
relation to fomiliar, everyday 
knowledge. Draw on less 
common knowledge. Infer factors 
to be considered. 


Level 4 


Locate and organize several 
pieces of embedded information, 
typically in a text whose content 
and form are unfamiliar. 


Construe the meaning of nuances 
of language in a section of text 
by taking into account the text as 
a whole. Show understanding 
and apply categories in an 
unfamiliar context. 


Critically evaluate a text or 
hypothesize about information in 
the text, using formal or public 
knowledge. Demonstrate an 
accurate understanding of long or 
complex texts. 


Level 5 


Locate and organize several 
pieces of information in 
unfamiliar contexts, where some 
information is deeply embedded 
and its relevance must be inferred 
from the text. 


Demonstrate a full and detailed 
understanding of a text whose 
content or form is unfamiliar. Deal 
with concepts that are contrary to 
expectations. 


Critically evaluate or hypothesize 
about the content of texts, 
drawing on specialized 
knowledge. Deal with concepts 
that are contrary to expectations. 




NOTE: The Program for International Student Assessment (PISA) uses five levels of performance to describe student performance. In 
order to reach a particular level, a student must be able to correctly answer a majority of items at that level. Students were classified into 
reading levels according to their scores. 

SOURCE: Organization for Economic Cooperation and Development, Program for International Student Assessment (PISA) 2000. 
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What is the relationship between the first and 
second parts of the sentence? 

The second part 

A. contradicts the first part. 

B. repeats the first part. 

C. illustrates the problem described in the 
first part. 

D. gives the solution to the problem 
described in the first part. 

More difficult tasks for reflecting on texts 
involve comparisons between something in the 
text and some element drawn from the reader’s 
own experience, knowledge, or ideas. The most 
difficult tasks involve the synthesis of elements 
derived from both the text and outside 
knowledge. A level 4 reflecting task (shown in 
appendix 4) shows an information sheet on flu 
immunization. Students are then asked to 
describe why part of the text may be misleading, 
and must respond by critically evaluating the text 
in terms of potential contradictions 
or exaggerations. 

Reading Literacy by Levels 

The percentage of 15-year-olds at each level of 
reading literacy is shown in figure 8. In addition, 
figure 9 (page 20) shows the percentages of 15- 
year-olds at the highest and lowest levels of the 
combined reading literacy scale. 

Overall, percentages of U.S. students across the 
levels are similar to the OECD average 
percentages, except at level 5. Twelve percent of 
U.S. 15-year-olds read at level 5, a percentage 
higher than the OECD average (table A3. 5). 
Looking across the countries, the U.S. proportion 
of students at level 5 is greater than that in 14 
countries and similar to that in 14 countries. 

At level 4, the United States has 21 percent of 
students, compared to 3 percent in Brazil and 32 
percent in Finland. Relative to U.S. 15-year-olds, 
five nations have higher percentages of their 
students reading at this level (Canada, Finland, 
Ireland, Japan, and Korea) and five nations have 
lower percentages of their 15-year-olds showing 
reading skills at level 4. Approximately one- 
quarter of U.S. students (27 percent) read at level 



3, similar to the OECD average of 29 percent. 
Another 21 percent read at level 2, again similar 
to the OECD average percentage. 

Twelve percent of 15-year-olds in the United 
States score at level 1, a percentage not different 
from that in 22 other nations or the OECD 
average. In other words, over two-thirds of all 
PISA 2000 participating countries have about the 
same percentages of students in level 1. Another 
6 percent of 15-year-olds are below level 1 in the 
United States. 

Looking at figure 8, one can see that in 
comparison to the United States, a few countries 
have large percentages of students at the highest 
levels and smaller numbers at the lowest levels 
(for example, Finland and New Zealand); a few 
countries have small percentages of students at 
the highest levels and larger numbers of students 
at the lowest levels (for example, Brazil and 
Mexico); and a few countries have large 
percentages of students at the middle levels and 
small percentages at either the lowest or highest 
levels (for example, Korea). Remaining countries 
like the United States, have a majority of 
students at levels 2, 3, and 4, and relatively 
balanced percentages of students at the highest 
and lowest levels. 

Another way to think about the levels is to 
consider not just the percentages of students at 
each particular level, but also to think about the 
cumulative percentages from level to level. For 
example, about one-third of U.S. students read at 
the two highest levels, level 4 or above. In Brazil, 
4 percent of students perform at levels 4 or 
above, and in Finland, about half of the students 
do. About 60 percent of students in the United 
States perform at level 3 or. above, and over 80 
percent at level 2 or above. Finland, with a 
higher average combined literacy score than the 
United States, has 79 percent of students reading 
at level 3 or above, and 93 percent of students 
reading at level 2 or above. 

Specific Skill Areas of Reading 
Literacy by Level 

Specific measures tap the three defined processes 
of reading literacy described earlier: retrieving 
information , interpreting texts , and reflecting on 
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Figure 8. — Percentage distribution of 1 5-year-olds by combined reading literacy scores, by level and by 
country: 2000 
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New Zealand 
Finland 
Australia 
Canada 
United Kingdom 
Ireland 
United States 
Belgium 
Norway 
Sweden 
Japan 
Switzerland 
Iceland 
Austria 
Germany 
France 
Denmark 
Czech Republic 
Poland 
Korea, Republic of 
Italy 
Flungary 
Greece 
Spain 
Portugal 
Luxembourg 
Mexico 

OECD average 

Non-OECD countries 

Liechtenstein 

Latvia 

Russian Federation 
Brazil 





Percent 



P71 Below level 1 Q Level 1 Q Level 2 | Level 3 Q Level 4 Q] Level 5 



NOTE: The Program for International Student Assessment (PISA) uses five levels of performance to describe student performance. In 
order to reach a particular level, a student must be able to correctly answer a majority of items at that level. Students were classified into 
reading levels according to their scores. Although the Netherlands participated in PISA in 2000, technical problems with its sample 
prevent its results from being discussed here. For information on the results for the Netherlands, see OECD (2001). The OECD average 
is the average of the national averages of 27 OECD countries. Because PISA is principally an OECD study, the results for non-OECD 
countries are displayed separately from those of the OECD countries and not included in the OECD average. 

SOURCE: Organization for Economic Cooperation and Development, Program for International Student Assessment (PISA) 2000. 
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Figure 9. — Percentage of 15-year-olds scoring at levels 1 and 5 on the combined reading literacy scale, 
by country: 2000 
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NOTE: The Program for International Student Assessment (PISA) uses five levels of performance to describe student performance. In 
order to reach a particular level, a student must be able to correctly answer a majority of items at that level. Students were classified into 
reading levels according to their scores. Although the Netherlands participated in PISA in 2000, technical problems with its sample 
prevent its results from being discussed here. For information on the results for the Netherlands, see OECD (2001). The OECD average 
is the average of the national averages of 27 OECD countries. Because PISA is principally an OECD study, the results for non-OECD 
countries are displayed separately from those of the OECD countries and not included in the OECD average. 

SOURCE: Organization for Economic Cooperation and Development, Program for International Student Assessment (PISA) 2000. 
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texts. Percentages of students performing at each 
level for each of these reading process subscales 
are shown in tables A3. 6, A3.7, and A3. 8. 

Looking across the countries, at almost every 
level on each reading literacy subscale, U.S. 
percentages of 15-year-olds are similar to the 
OECD average. 11 In general, few countries 
perform differently than the United States at any 
given level for each of the three scales, leaving a 
large group of countries that perform similarly to 
the United States. 

For example, on the retrieving information 
subscale, five countries have greater percentages 
of 15-year-olds at level 5 than the United States, 
while eight countries have lower percentages, 
ranging from less than 1 percent in Brazil to 26 
percent in Finland. That leaves 17 countries with 
percentages at level 5 similar to the U.S. 
percentage. At the opposite end of the scale, at 
level 1 for the retrieving information scale, three 
countries have a smaller proportion of students 
performing at this level than the United States 
and seven have a greater proportion. 

On the interpreting scale, two countries have 
higher percentages of students at level 5. At the 
other end of the interpreting scale, three 
countries have a smaller proportion of 15-year- 
olds at level 1. 

The U.S. has a greater percentage of students at 
level 5 on the reflecting scale than 10 countries, 
while 3 countries have greater percentages than 
the United States at the same level. The United 
States has a higher percentage of students at level 
1 than four countries on the same reflecting scale. 
At levels 2, 3, and 4 on the reflecting scale the 
U.S. has similar percentages to the OECD 
average. At least two-thirds of countries are 
similar to the U.S. percentages at these levels. 

Although patterns of scores for the reading 
literacy subscales vary across countries, generally 
within each country proportions of 15-year-olds 



at each level are similar across all three reading 
literacy subscales, as well as the combined 
reading literacy scale. In the United States, for 
instance, about 12 percent of students are at level 

5 for retrieving information , interpreting texts , 
reflecting on texts , and for the combined reading 
literacy scale. In fact, the percentage of U.S. 15- 
year-olds across the three dimensions and the 
combined reading literacy scale is consistent at 
each level: about 21 percent for retrieving 
information , interpreting texts , and reflecting 
on texts at level 4; about 27 percent for each 
scale at level 3; about 21 percent at level 2 for 
each scale; about 12 percent at level 1; and about 

6 percent below level 1. That is, there is no 
difference between percentages of U.S. 

15-year-olds performing at high and low levels 
for the retrieving information, interpreting texts, 
or reflecting on texts scales. 

Linking Performance by Levels to 
PISfl Items 

What do these U.S. results mean, then, in terms 
of the kinds of reading literacy questions PISA 
asks? Given that about 92 percent of U.S. 
students perform at level 1 and above on the 
retrieving information scale, a large proportion 
of students should be able to answer the level 1 
question described above (see also appendix 4) 
about why athletic shoes should not be too rigid. 
Indeed, 84 percent of U.S. students answer this 
question correctly. On average 80 percent of 
students in OECD countries answer this question 
correctly. About 94 percent of U.S. students also 
perform at level 1 and above on the interpreting 
and reflecting on texts scales, so similarly large 
proportions of students should answer the level 1 
question about how parts of a sentence from the 
athletic shoe article relate to one another correctly 
as well as the question about the author’s intent. 
Seventy-eight percent of U.S. students do answer 
correctly the reflecting item about how parts of a 
sentence relate to one another, and 78 percent 
also answer correctly the interpreting item about 
author’s intent. 



11 At level 5 for the interpreting texts scale, the U.S. percentage is higher than the OECD average. 
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In the same way, for the level 4 retrieving item 
described above in which students had to locate 
two characters on stage using dialogue and stage 
directions from a play, about one-third of U.S. 
15-year-olds (those at levels 4 and 5) should be 
able to answer this question a majority of the 
time. Actual responses show that 38 percent of 
U.S. 15-year-olds answer this item correctly, and 
an average of 47 percent of OECD students 
answer correctly. Forty-one percent of U.S. 
students also receive full credit for answering the 
level 4 reflecting item (using the flu immunization 
sheet, also described above and included in 
appendix 4) correctly. About 13 percent of U.S. 
students perform at level 5 on the interpreting 
scale, and indeed, 15 percent of U.S. students 
receive full credit for a level 5 interpreting item 
using the labor force structure diagram (see 
appendix 4). An average of 14 percent of OECD 
students receive full credit for the same item. 



Summary 

The results in this chapter show that, in general, 
U.S. students perform similarly to the OECD 
average, both for the combined reading literacy 
scale and in the percentages of students who 
perform at each level on the specific reading 
subscales of retrieving information , interpreting 
texts , and reflecting on texts. Numbers of high 
performing students meeting international 
benchmarks are also similar to the OECD 
benchmarks, as is variation in student 
performance. The next chapter will discuss 
results of PISA 2000 for mathematics literacy 
and science literacy. 
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o In both mathematics and science literacy, the United States average does not 
differ from the OECD average. Eight countries outperform the United States 
in mathematics literacy, and seven have higher average scores for science 
literacy. The United States has higher average scores than seven countries for 
mathematics literacy and seven for science literacy (figure 10; table A3. 9). 

o The top 10 percent of students in OECD countries score 625 or higher in 
mathematics literacy. In the United States, 9 percent of students score at this 
level or better, a percentage not different from the OECD top 10 percent 
benchmark. In eight countries, a greater proportion of students score in the 
top 10 percent, while six countries have a smaller proportion (figure 13; 
table A3. 12). 

© For science literacy, the top 10 percent of all students score 627 or higher. 

In the United States, 10 percent of students score at this level or better. Four 
countries have a higher percentage of students score in the top 10 percent, 
while seven countries have a lower percentage (figure 14; table A3. 13). 
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In addition to assessing students in reading 
literacy, PISA 2000 tested the mathematics and 
science literacy of 15-year-olds in participating 
countries. As with reading literacy, PISA’s 
mathematics and science literacy assessments 
focus on 15-year-olds’ ability to apply 
mathematical and scientific principles and 
thinking in a wide variety of situations. As 
societies deal with changes and advances in 
technology, in medicine, in the environment, and 
in economic situations, mathematics literacy and 
science literacy are important skills for 
understanding and managing those changes. 

As mentioned previously in this report, PISA 
2000 concentrates on reading literacy, devoting 
two-thirds of testing time to this subject. The 
testing time remaining for the areas of 
mathematics and science literacy limits the 
number of items covering these topics in the 
assessment. As a result, a single scale is reported 
for each of mathematics and science literacy. 
Levels are not defined for either scale. The 
mathematics and science literacy scales are 
comparable to the combined reading literacy 
discussed earlier. 

In discussing the results from the mathematics 
and science literacy portions of PISA 2000, it is 
important to understand the concept of 
mathematics and science literacy utilized by PISA. 
The following sections provide PISA’s definitions 
of mathematics and science literacy and place 
these definitions in the context of the PISA 2000 
assessments. As with the preceding chapter on 
reading literacy, this chapter begins by discussing 
national averages in mathematics and science 
literacy, and then examines score distributions 
within countries and across countries by using 
national and international benchmarks. 

Examples of mathematics and science literacy 
items are also presented. 

Defining Mathematics Literacy 

Just as reading literacy means more than the 
ability to read and write, mathematics literacy 
means more than the ability to add and subtract 



or perform other mathematical computations. 
PISA defines mathematics literacy as: 

the capacity to identify, to understand the 
role that mathematics plays in the world, 
to make well-founded mathematical 
judgments and to engage in mathematics, 
in ways that meet the needs of an 
individual's current and future life as a 
constructive, concerned and reflective 
citizen (OECD 1999, p.41). 

As noted earlier, literacy in all three areas of 
reading, mathematics, and science is defined in 
terms of three dimensions — content, process, and 
situation. For mathematics literacy, the three 
dimensions are specified as follows: 

«e> Content or Structure — includes broad 

mathematical concepts or content underlying 
mathematical thinking: chance, change and 
growth, space and shape, reasoning, 
uncertainty and dependency relationships. 

In PISA 2000, change and growth and space 
and shape were emphasized. 

Process — consists of thinking skills organized 
into three competency classes: simple 
computations or definitions; connections to 
be made to solve straightforward problems; 
and mathematical thinking, generalization, 
and insight (reflection) requiring students to 
analyze, identify mathematical elements in a 
given situation, and suggest new problems. 

® Situation — comprises the situations in which 
mathematics knowledge and skills are 
applied, classified as: private life/personal, 
school life, work and sports, local 
community and society, and scientific. 

In sum, PISA mathematics literacy items seek to 
measure how well students are able to apply a 
variety of mathematical processes to an 
assortment of problems — the kind of problems 
they solve for a school assignment; the kind they 
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solve to help make home improvements; and the 
kind they encounter while reading a newspaper 
or magazine. 

Defining Science Literacy 

In the broad conception used in PISA, science 
literacy means the ability to think scientifically, 
understand some specific scientific concepts, and 
take a scientific approach to problem solving. In 
this sense, science literacy is defined as: 



As with the other PISA items, science literacy 
items seek to measure how well students are able 
to apply a variety of scientific skills to a diverse 
group of situations — the kind of situations they 
encounter while making a decision about an 
environmental referendum; the kind they 
encounter in trying to understand their own 
medical care or while reading information about 
disease prevention; or the kind they encounter in 
a scientific text. These definitions of mathematics 
and science literacy provide a basis for the 
following presentation of results from PISA 2000. 



the capacity to use scientific knowledge , to 
identify questions , and to draw evidence - 
based conclusions in order to understand 
and help make decisions about the natural 
world and the changes made to it through 
human activity (OECD 1999, p.60). 

The specifics of the three defining dimensions — 
content, process, and situation — in the case of 
science literacy are as follows: 

o Content or Structure — involves broad 
science concepts from physics, chemistry, 
biological sciences, and Earth and space 
sciences. Concepts are incorporated more 
particularly from themes such as biodiversity ; 
forces and movement, and physiological 
change , and are organized into several broad 
areas of application: science in life and 
health, science in Earth and environment, 
and science in technology. 

o Process — includes thinking skills organized 
into five processes: recognizing scientifically 
investigable questions, identifying evidence 
needed in a scientific investigation, drawing 
or evaluating conclusions, communicating 
valid conclusions, and demonstrating 
comprehension of scientific concepts. 

o Situation — focuses on the situations in which 
scientific knowledge and skills are applied: 
personal, public, global, and situations of 
historical relevance. 
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Mathematics and Science 
Literacy of PISA Countries 

National averages provide the simplest 
description of performance for mathematics and 
science literacy. In PISA 2000, national averages 
in mathematics and science literacy are strongly 
correlated with each other, and with national 
averages in reading literacy. This is not 
surprising, considering that both PISA 
mathematics and science literacy items focus on 
the application of mathematical and scientific 
thinking to real-world situations, which often 
entails interpreting charts or other documents. 
Some items, particularly in science, also require 
significant amounts of reading. The United 
States’ relative position compared to its 
international peers is approximately the same in 
both mathematics and science, and the countries 
scoring above and below the United States are 
largely the same for both subjects. More 
countries score higher than the United States in 
mathematics literacy and in science literacy than 
on the combined reading literacy scale. With this 
situation in mind, the discussion that follows 
considers mathematics and science literacy 
simultaneously. Departures from this pattern are 
highlighted. 

In figure 10 (page 26), shading identifies 
countries whose averages are higher, lower, or not 
different from that of the United States in 
mathematics and science literacy. Non-OECD 
countries are shown at the bottom of the table 
with shading to indicate differences from the 
United States; however, non-OECD countries are 
not included in determining the OECD average. 
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Figure 1 0. — Mathematics and science literacy average scores of 1 5-year-olds, by country: 2000 



T 

E 

R 



KD 

KD 



KD 

KD 

KS> 

KD 

KD 

KD 



Mathematics literacy 



Science literacy 



Country 



Japan 

Korea, Republic of 
New Zealand 
Finland 
Australia 
Canada 
Switzerland 
United Kingdo 



Belgium 

France 

Austria 

Denmark 

Iceland 

Sweden 

Ireland 

Norway 

Czech Republic 

United States 

Germany 

Hungary 

Spain 

Poland 



OECD average 



Non-OECD countries 
Liechtenstein 
Russ ian Federation 



520 

517 

515 

514 

514 

510 

503 

499 

498 

493 

490 

488 

476 

470 



500 



Average 




Country 


Average 


Korea, Republic of 


552 


Japan 


550 


Finland 


538 


United Kingdom 


532 


Canada 


529 


New Zealand 


528 


Australia 


528 


Austria 


519 


Ireland 


513 


Sweden 


512 


Czech Republic 


511 


France 


500 


Norway 


500 


United States 


499 


Hungary 


496 


Iceland 


496 


Belgium 


496 


Switzerland 


496 


Spain 


491 


Germany 


487 


Poland 


483 


Denmark 


481 


Italy 


478 


Greece 


461 


Portugal 


459 


Luxembourg 


443 


Mexico 


422 


OECD average 


500 


Non-OECD countries 


Liechtenstein 


476 




I | Average is significantly higher than the U.S. average 
I | Average is not significantly different from the U.S. average 
Average is significantly lower than the U.S. average 



NOTE: Although the Netherlands participated in the Program for International Student Assessment (PISA) in 2000, technical problems 
with its sample prevent its results from being discussed here. For information on the results for the Netherlands, see OECD (2001). The 
OECD average is the average of the national averages of 27 OECD countries. Because PISA is principally an OECD study, the results for 
non- OECD countries are displayed separately from those of the OECD countries and not included in the OECD average. 

SOURCE: Organization for Economic Cooperation and Development, Program for International Student Assessment (PISA) 2000. 
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As noted earlier, both assessments generated a 
single literacy score between 0 and 1,000 (with 
most scores falling between 200 and 800) for 
each student in each of the subjects. Country 
averages constructed from these scores provide 
the basis to compare countries in each subject. 

In both mathematics and science literacy, the 
United States’ average does not differ from the 
OECD average. Compared to the U.S. average, 
eight countries have higher average scores in 
mathematics literacy, and seven countries have 
higher scores in science literacy. The same seven 
countries appear in both groups — Australia, 
Canada, Finland, Japan, Korea, New Zealand, 
and the United Kingdom. For mathematics 
literacy, the average score for Switzerland is also 
higher than the U.S. average score. In 
mathematics literacy, 15 countries score similarly 
to the United States; in science literacy that 
number is 16 nations. The United States 
performs better than seven countries for 
mathematics literacy and seven countries for 
science literacy. Six of the same countries appear 
in both of those groups — Brazil, Greece, Latvia, 
Luxembourg, Mexico, and Portugal. In addition, 
the United States outperforms Italy in 
mathematics literacy and the Russian Federation 
in science literacy. 

Despite differences in the frameworks for 
mathematics and science in TIMSS 1999 and 
PISA and a difference in the age of students being 
assessed, some countries with average scores 
higher than the United States in TIMSS 1999 also 
have higher average scores in PISA 2000. For 
example, Australia, Canada, Japan, and Korea 
have higher average scores than the United States 
in PISA 2000 for mathematics and science 
literacy, and also have higher scores for 
mathematics and science in TIMSS 1999. New 
Zealand, which has an average score not 
significantly different from the United States score 
for mathematics and science on TIMSS 1999, has 
a higher average score for mathematics and 
science literacy on PISA 2000. 
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The Distribution of Mathematics 
and Science Literacy 

National Percentiles 

Average scores for mathematics and science 
literacy, while providing a concise way to describe 
performance, provide no information about the 
way scores are distributed in countries. This 
section, as with the previous discussion of reading 
literacy, looks at national percentile scores and 
score distributions in order to provide some 
information about how the scores that make up 
each country’s average score vary. 

Examining selected score cut points (for the 5th 
and 95th percentiles, in this case) within countries 
provides one means to compare distributions 
across countries. For example, to be among the 
top 5 percent of scorers in Brazil requires a score 
of 499 or better, in the United States it requires a 
score of 652 or better, and in New Zealand it 
requires a score of 689 or better. The cutoff point 
to identify the bottom 5 percent of scorers begins 
at 179 in Brazil, 327 in the United States, and 
402 in Japan. 

It is also possible to compare the U.S. average 
score to corresponding cut points in several 
countries. Figure 11 (page 28) shows graphically 
that while the average U.S. mathematics literacy 
score is 493, that is the 25th percentile score in 
Korea. This means that 75 percent of Korean 
students score at or above the U.S. average. 
Similarly, a score of 493 is just under the 90th 
percentile score in Mexico, which means that 
about 10 percent of Mexican students perform at 
or above the U.S. average for mathematics literacy. 

An examination of the size of the score point 
difference between the top 5 percent and bottom 
5 percent of scorers in a country also gives a 
sense of the variability of scores. The bottom 5 
percent of U.S. students score 327 or less, but the 
top 5 percent score 652 or better, which means 
that there is at least a 325 point difference 
between the top 5 percent and bottom 5 percent 
of 15-year-olds for mathematics literacy in the 
United States. The length of the bars in figure 1 1 
gives a visual indication of these kinds of 
differences in scores between a country’s highest 
and lowest performing students. 
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Figure 1 1 . — Distribution of mathematics literacy scores of 15-year-olds by percentiles, by country: 2000 
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NOTE: Although the Netherlands participated in the Program for International Student Assessment (PISA) in 2000, technical problems 
with its sample prevent its results from being discussed here. For information on the results for the Netherlands, see OECD (2001). The 
OECD average is the average of the national averages of 27 OECD countries. Because PISA is principally an OECD study, the results for 
non-OECD countries are displayed separately from those of the OECD countries and not included in the OECD average. 

SOURCE: Organization for Economic Cooperation and Development, Program for International Student Assessment (PISA) 2000. 
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It is not accurate to rank countries’ scoring 
variation based on the length of these bars 
because of the statistical techniques used to 
sample students, but standard deviations of the 
scores give a technical way to examine 
.^variations in scores within a country to learn 
how greatly scores within a country differ from 
their average scores (data not shown; see table 
A3. 10). For mathematics literacy, seven 
countries have less variation in performance 
than the United States. No country has greater 
variation in performance than the United States, 
but U.S. variation in performance is similar to 
that in 23 other countries. It is also similar to 
the OECD average variation. 

For science literacy, the story is similar to 
mathematics literacy. Looking at the U.S. average 
for science literacy in comparison to selected 
cutpoints for other countries shows differences in 
distributions of scores (figure 12, page 30). The 
average score for science literacy in the United 
States is 499, and, as with mathematics literacy, 
about 75 percent of Korean 15-year-olds score 
above the U.S. average. That is, the U.S. average 
for science literacy corresponds to the 25th 
percentile in Korea. The U.S. science literacy 
average also corresponds to approximately the 
90th percentile in Mexico, showing that about 10 
percent of Mexican 15-year-olds perform above 
the U.S. average for science literacy. 

Again, the figure gives a sense of the spread of 
scores for each country. In the United States, the 
top 5 percent of 15-year-olds score at or above 
658, while the bottom 5 percent score at or 
below 330, a difference of 328 points at a 
minimum. Comparing standard deviations of the 
average scores for science literacy gives a 
statistically accurate way to describe variation in 
performance, and shows that, as with 
mathematics literacy, a few countries (6 countries) 
have less variation in performance than the 
United States (data not shown; see table A3. 11). 
The majority of countries (24 countries) show 
similar variation in performance to the United 
States, and the U.S. variation is similar to the 
OECD average. 




International Percentiles 

Scores for 15-year-olds from each PISA 2000 
participating country can also be analyzed by 
sorting them into groups that meet international 
benchmarks — in this case, proportions of students 
from each country who are among the top 10 
and top 25 percent of students from all countries 
combined. This provides another means to 
examine how scores are distributed across 
countries by showing which countries have 
greater proportions of students performing at 
high levels internationally. 

The top 10 percent of students in OECD 
countries score 625 or higher in mathematics 
literacy. In the United States, 9 percent of 
students achieve this score or better, a percentage 
not different from the OECD top 10 percent 
benchmark. Eight countries have a greater 
proportion of students score in the top 10 
percent, while six countries have a smaller 
proportion (figure 13, page 31; table A3. 12). 

Twenty-two percent of U.S. 15-year-olds meet the 
top 25 percent international benchmark, which 
requires a score of 571 or better. Nine countries 
have higher percentages at this benchmark, and 
six have lower percentages. The same eight 
countries that have higher percentages of students 
in the top 10 percent than the United States also 
have a greater percentage in the top 25 percent, 
with the addition of Belgium for this benchmark. 

Figure 13 shows that Japan, for instance, has the 
same percentage of students (22 percent) reaching 
the top 10 percent benchmark as the United 
States has of students reaching the top 25 percent 
benchmark (also 22 percent). This means that 22 
percent of Japanese students score at or above 
625 in mathematics literacy, compared to 9 
percent of U.S. students. Similarly, Mexico has 
about the same percentage of students in the top 
50 percent (8 percent; data not shown, see table 
A3. 12) as the United States has in the top 10 
percent. This means that about 8 percent of 
Mexican students score at or above 505 in 
mathematics literacy, compared to about 46 
percent of U.S. students. 
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Figure 1 2. — Distribution of science literacy scores of 1 5-year-olds by percentiles, by country: 2000 
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NOTE: Although the Netherlands participated in the Program for International Student Assessment (PISA) in 2000, technical problems 
with its sample prevent its results from being discussed here. For information on the results for the Netherlands, see OECD (2001). The 
OECD average is the average of the national averages of 27 OECD countries. Because PISA is principally an OECD study, the results for 
non-OECD countries are displayed separately from those of the OECD countries and not included in the OECD average. 

SOURCE: Organization for Economic Cooperation and Development, Program for International Student Assessment (PISA) 2000. 
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Figure 1 3. — Percentage of 1 5-year-olds reaching PISA international benchmarks in mathematics literacy, 
by country: 2000 
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NOTE: Although the Netherlands participated in the Program for International Student Assessment (PISA) in 2000, technical problems 
with its sample prevent its results from being discussed here. For information on the results for the Netherlands, see OECD (2001). 

SOURCE: Organization for Economic Cooperation and Development, Program for International Student Assessment (PISA) 2000. 
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For science literacy, the top 10 percent of all 
students score 627 or higher. In the United 
States, 10 percent of students achieve this score 
or better. Four countries have a higher 
percentage of students in the top 10 percent, 
while seven countries have a lower percentage 
(figure 14; table A3. 13). 

Four of the eight countries with higher 
percentages of students in the top 10 percent than 
the United States for mathematics literacy (Japan, 
Korea, New Zealand, and the United Kingdom) 
also have higher percentages than the United 
States for science literacy. In both cases, Japan 
has about double the percentage of students in 
the top 10 percent than the United States. 

Twenty-five percent of U.S. students score at or 
above the cut point score of 572 for the top 25 
percent benchmark in science literacy. Seven 
countries have a higher percentage, and seven 
have lower percentages. The same four 
countries (Japan, Korea, New Zealand, and the 
United Kingdom) that have higher percentages 
of students in the top 10 percent than the 
United States for science literacy also have more 
students in the top 25 percent, with the 
addition of Australia, Canada, and Finland for 
this benchmark. 

Four countries (Japan, Korea, New Zealand, and 
the United Kingdom) have more students meeting 
both the 10 percent and 25 percent benchmark in 
mathematics and science literacy than the United 
States. In general, however, more countries have 
higher percentages of students meeting the 
international benchmarks than the United States 
for mathematics literacy than science literacy. 

The United States percentages for both the 10 
percent and 25 percent benchmarks in 
mathematics and science literacy are consistent. 



Examples of Mathematics Items 

One way of developing a deeper understanding of 
what types of mathematics questions are asked of 
students in PISA 2000 is to examine actual 
mathematics items from the assessment. Because 
mathematics literacy is a minor area in the 2000 
assessment, there are not enough released items at 
this point in time to show the broad range of 
content, process, and situations students are 
presented with in PISA 2000. However, the items 
described below do provide a glimpse into the 
mathematics component of PISA.2000. 

As is the case of reading literacy, PISA 
mathematics items are divided into units that 
contain a short text or picture, and several 
questions or items that require the use of the text 
or picture to answer. For example, in the unit 
“Continent Area” students are provided with a 
map of Antarctica and a companion scale, and 
are asked to estimate the area of the continent 
using the map scale. This item and other 
mathematics and science items are shown in 
appendix 4. Students are also asked to show 
their work and explain how they made their 
estimate. Full credit answers require students to 
provide a correct estimate. The correct estimate 
is from 12 million to 18 million square 
kilometers. Students show evidence of using a 
correct method through drawing a square, 
rectangle, or circle around the continent and 
using those dimensions to estimate the area. In 
addition, students can add the areas of several 
regular geometric figures they draw on the map. 
Students who provide a correct estimate and 
show evidence of other correct methods, or just 
provide the correct estimate, also receive full 
credit. This question corresponds to a score of 
722 on the mathematics literacy scale. 12 This 
means that a student with a score of 722 or 
higher could answer this question a majority of 
the time. This is above the score of 625 at which 
the top 10 percent of all students score. Ten 
percent of U.S. students receive full credit for the 



12 



PlSA’s items are scaled such that for an item corresponding to a score of 722, students who score 722 on that scale will have a 62 
percent probability of answering the item correctly. Students with a score above 722 will have a higher probability of answering the 
item correctly, and students with scores lower than 722 will have a lower probability of answering the item correctly. 
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Figure 14, — Percentage of 15-year-olds reaching PISA international benchmarks in science literacy, by 
country: 2000 




Top 1 0 percent 




Top 25 percent 




„ Country 1 


Percent 


Country 


Percent 


Japan 


20 


Japan 


43 


Korea, Republic of 


18 


Korea, Republic of 


43 


United Kingdom 


17 


Finland 


36 


New Zealand 


16 


New Zealand 


36 


Finland 


15 


United Kingdom 


35 


, Australia 


15 


Australia 


34 . 


Canada 


14 


Canada 


33 


Austria 


11 


Austria 


30 


Czech Republic 


11 


Sweden 


28 


Belgium 


11 


Ireland 


27 


Ireland 


11 


Belgium 


27 


France 


11 


Czech Republic 


27 


Sweden 


11 


France 


26 


Hungary 


10 


United States 


25 


United States 


10 


Hungary 


24 


Switzerland 


10 


Norway 


24 


Norway 


8 


Switzerland 


23 


Germany 


8 


Germany 


21 


Denmark 


8 


Spain 


■ 21 


Spain 


7 


Iceland 


20 


Poland 


7 


; Denmark 


20 


Iceland 


6 , 


Poland 


19 


Italy 


6 ! 


Italy 


17 


Greece 


4 


Greece 


13 


Portugal 


3 


Portugal 


11 


Luxembourg 


2 


Luxembourg 


8 


Mexico 


1 


Mexico 


3 



Non-OECD countries Non-OECD countries 




j 1 Percentage is significantly higher than the U.S. percentage 
j 1 Percentage is not significantly different from the U.S. percentage 



j 1 Percentage is significantly lower than the U.S. percentage 




*The shading of Liechtenstein in the top 10 percent category may appear incorrect; however, statistically, its placement is correct. 

^Too small to report. 

NOTE: Although the Netherlands participated in the Program for International Student Assessment (PISA) in 2000, technical problems 
with its sample prevent its results from being discussed here. For information on the results for the Netherlands, see OECD (2001). 

SOURCE: Organization for Economic Cooperation and Development, Program for International Student Assessment (PISA) 2000. 
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question, with another 38 percent getting partial 
credit. On average in OECD countries, 20 
percent of students receive full credit and another 
40 percent receive partial credit for this question. 

This problem falls under the content category of 
space and shape , and requires students to 
demonstrate their mathematical skills and 
knowledge in a private life/personal situation. In 
terms of process, this item requires making 
connections to solve the problem. 

Another mathematics unit (“Triangles,” shown in 
appendix 4) asks students to determine which of 
five figures presented fits a specific geometric 
description. This item corresponds to a score of 
546 on the mathematics literacy scale. 13 In the 
United States, 46 percent of students answer this 
question correctly, while an average of 62 percent 
of students in OECD countries answer correctly. 

This question is also from the content category 
space and shape , and requires a computation 
process in a scientific situation. 

These items show, in a limited way, the kind of 
applied reasoning and real-life contexts that PISA 
seeks to employ in its assessment items. As 
noted, additional examples of items can be found 
in the OECD international report on PISA 2000 
Knowledge and Skills for Life— First Results 
from the OECD Programme for International 
Student Assessment and on the OECD PISA Web 
Site, www.pisa.oecd.org. The next cycle of PISA 
in 2003, in which mathematics literacy will be the 
major subject, will contain a much larger range of 
items in terms of content areas as well as 
processes and situations. 



Examples of Science Items 

Similar to the situation with mathematics, the 
number of released science literacy items from 
PISA 2000 is relatively small, and does not allow 
the opportunity to discuss the broad range of 
content, process, and situations students are faced 
with in the science literacy portion of PISA 2000. 
However, the following items are illustrative of 
some of the types of questions students are 
presented in PISA 2000. Again, the full text of 
these samples and others can be found in 
appendix 4. 

In one example, students are presented with a 
brief passage from an article on the ozone layer. 
Using the passage, students are asked to answer 
the following question: 

Lines 14 and 15 state: “Without this beneficial 

ozone layer, humans would be more susceptible 

to certain diseases due to the increased incidence 

of ultra-violet rays from the Sun.” 

Name one of these specific diseases. 

To receive full credit, students must refer to skin 
cancer (or melanoma). This question requires 
students to demonstrate understanding of 
scientific concepts in a global situation, and 
draw on their everyday knowledge of 
physiological change (science in life and health) 
related to sun exposure. This question 
corresponds to a score of 560 on the science 
literacy scale. 14 This is 10 points below the score 
at which the top 25 percent of all students can 
be identified. In the United States, 63 percent of 
students answer this question correctly; on 
average in the OECD, 63 percent of students 
also answer correctly. 



13 See footnote 12. 

14 See footnote 12. 5 Q 
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Another question based on the same ozone article 
asks: 

At the end of the text, an international meeting 
in Montreal is mentioned. At that meeting lots 
of questions in relation to the possible depletion 
of the ozone layer were discussed. Two of those 
questions are given in the following table. 

Which of the questions below can be answered 
by scientific research? 



As with the mathematics literacy items above, 
these examples show in a limited way the kind of 
applied reasoning and real-life contexts that PISA 
seeks to employ in its assessment items. Science 
literacy will be the main PISA subject in 2006. 
During that cycle, science literacy items will take 
on a greater role in PISA, and the breadth of 
items will be much greater than those used in 
PISA 2000. 



Circle Yes or No for each. 



Question: 


Aiiswerable 
by scientific 
research? 


Should the scientific uncertainties 
about the influence of CFCs on 
the ozone layer be a reason for 
governments to take no action? 


Yes / No 


What would the concentration of 
CFCs be in the atmosphere in the 
year 2002 if the release of CFCs 
into the atmosphere takes place 
at the same rate as it does now? 


Yes / No 



To receive full credit, students must indicate 
“No” for the first question and “Yes” for the 
second. This item required students to use their 
knowledge of science in the Earth and 
environment and to recognize scientific 
questions. This item corresponds to a score of 
542 on the science literacy scale. 15 Sixty-four 
percent of U.S. students respond correctly to this 
question, and an average of 59 percent of 
students in OECD countries respond correctly. 



15 See footnote 12. 
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Summary 

The results presented in this chapter show that, 
in general, the United States performs similarly 
to the OECD average for both mathematics 
literacy and science literacy. However, more 
countries outperform the United States for 
mathematics and science literacy than for reading 
literacy. At the same time, more countries have 
similar variation to the United States in 
mathematics and science literacy (23 countries 
and 24 countries, respectively) than for reading 
literacy, where 15 countries are similar to the 
United States. This suggests that while more 
countries have less variation in average scores 
than the United States for reading literacy, this 
does not necessarily mean that they perform 
better than the United States. 

Percentages of U.S. students meeting international 
benchmarks are also similar to OECD 
percentages, showing a distribution similar to 
most OECD countries. A few countries have 
higher percentages of students meeting top 
benchmarks in several areas. For example, New 
Zealand has higher percentages of students in the 
top 10 percent than the United States for reading 
literacy, mathematics literacy, and science literacy. 
Korea, in contrast, has a lower percentage of 
students in the top 10 percent for reading literacy 
than the United States, but a higher percentage 
for mathematics and science literacy. These 
percentages help illustrate in which areas different 
countries perform particularly well compared to 
the United States. 

The next chapter will examine how different 
subgroups of students — such as males and 
females, students with different language 
backgrounds, students with parents of varying 
educational backgrounds, and different racial 
and ethnic groups in the United States — perform 
on reading, mathematics, and science literacy in 
PISA 2000. 
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Demographic Profiles of 
Reading, Mathematics, 
and Science Literacy 



Key Findings 

o On the combined reading literacy scale, females outperform males in every 
country. On the PISA 2000 mathematics literacy assessment, performance of 
males and females in the United States is similar, as it is in 16 other countries. 
For most countries (26 out of 31 countries), including the United States, males 
and females perform similarly on the science literacy assessment (figure 15; 
tables A3. 14, A3. 18, and A3. 19). 

o In the United States, parents’ education is strongly linked to differences in 
student performance in reading, mathematics, and science literacy (figure 16; 
tables A3. 20, A3.21, and A3.22). 



o In the United States, increases in socioeconomic status are associated with 
increases in scores for reading literacy, mathematics literacy, and science literacy. 
Most participating countries do not differ significantly from the United States in 
terms of the strength of the relationship between socioeconomic status and 
literacy in any subject (figure 17; table A3. 23). 

o In the United States, parents’ national origin is linked to performance in reading 
literacy and mathematics literacy only for those students with two foreign-born 
parents compared with students with two native-born parents. There was no 
difference in science literacy achievement between students with native and 
foreign-born parents (figure 18; tables A3 .24, A3. 25, and A3. 26). 

o In the United States and most other countries, the reading literacy achievement of 
students who speak the test language at home is higher than that of students not 
speaking this language at home. The United States and most other countries also 
show advantages for test-language speakers in mathematics and science literacy 
(figure 19; tables A3.27, A3.28, and A.3.29). 
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o The pattern of between-group differences for racial and ethnic groups in the 
United States is identical across the three literacy areas. In reading, mathematics, 
and science, the average literacy scores for Whites and “other” students are 
higher than for Hispanic and Black students (figure 20; table A3. 30). 
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In the United States and many other countries, 
policymakers are not only interested in overall 
achievement, but also in achievement by specific 
population groups. The preceding chapters have 
discussed differences in student performance, but 
not differences among students themselves and 
how these might relate to performance. This 
chapter focuses on the performance of various 
demographic groups within the population of 
students aged 15 years in each participating 
country. Differences in reading, mathematics, and 
science literacy are presented for gender, parents’ 
education, socioeconomic background, parents’ 
national origin, and language spoken at home. 
Countries differ not only in average performance, 
but also in the performance of these population 
groups, and in the extent to which these groups 
differ from each other. In addition, this chapter 
presents performance results for PISA 2000 by 
race and Hispanic origin for the United States. 

Gender 

Equity between males and females in educational 
opportunity is an important education policy goal 
in OECD countries, since it can have far-reaching 
consequences for economic opportunities and 
quality of life. Patterns of gender differences in 
student achievement by subject matter and across 
countries can point to areas of strength and 
weakness within educational systems seeking to 
provide equal access to learning for both males 
and females. 

Previous studies have shown that, on average, 
females tend to perform better on reading 
assessments, and males tend to perform slightly 
better in mathematics and science, particularly at 
higher grade levels (Donahoe et al. 2000; 

Braswell et al. 2001; Mullis et al. 2000; Gonzales 
et al. 2000). Gender differences in mathematics 
and science achievement are more variable than 
in reading achievement across assessments, 
countries, and grades (Elley 1992, Mullis et al. 
2000). The Third International Mathematics and 



Science Study (TIMSS) found a gender gap 
favoring males in mathematics and science 
achievement in more countries among eighth- 
graders than among fourth-graders, and the 
differences were more pronounced in eighth grade 
(Mullis et al. 2000). 

The differences between the assessment scores of 
males and females for each country in each of the 
three subjects assessed by PISA are shown in 
figure 15. A bar extending under the females’ side 
of the center divide indicates that females 
outperform males in that country, and vice versa. 
The size of the bar represents the difference in 
score points between males and females. The 
color of the bar indicates whether this difference 
is statistically significant: a dark bar indicates a 
significant difference; a light bar does not. For 
example, on the combined reading literacy scale 
in Australia, females outperform males by over 
30 points on average, and that difference is 
statistically significant. 

On the combined reading literacy scale, females 
outperform males in each of the 31 nations for 
which results are presented here. With few 
exceptions, the same holds true for performance 
on the three process subscales on the reading 
assessment (retrieving information , interpreting 
texts y and reflecting on texts ; data not shown, 
see tables A3. 15, A3. 16, and A3. 17). 16 

The size of the difference between females and 
males in reading literacy in the United States is 
similar to that of most other OECD nations, with 
the exception of Finland, where the gap on the 
combined reading literacy scale is larger than that 
of the United States. Finland’s gender gap is 
greater than that of the United States not only on 
the combined reading literacy scale but in each of 
the three reading process subscales as well (data 
not shown; see tables A3. 15, A3. 16, and A3. 17). 
The difference between combined reading literacy 
scores for males and females in Finland is 51 
points, and the difference in the United States is 
29 points. In addition, the gap between females’ 



16 In Brazil, Liechtenstein, and Mexico the gender difference is not significant for the retrieving information dimension, and for Korea 
the difference for interpreting texts is also not statistically significant. 
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Figure 1 5. — Differences in average scores in reading, mathematics, and science literacy of 1 5-year-olds 
by gender, by country: 2000 
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NOTE: Each bar above represents the average score difference between males and females on combined reading, mathematics or science 
literacy. Some of these differences are statistically significant and indicated by darker bars. For instance, the United States has a 29 point 
score difference favoring females in combined reading literacy, which is statistically significant. The score differences between U.S. males 
and females in mathematics literacy and science literacy are 7 points and 5 points, respectively, but neither is a statistically significant 
difference. Average score difference is calculated by subtracting scores of males from scores of females. Detail may not sum to totals due 
to rounding. Although the Netherlands participated in the Program for International Student Assessment (PISA) in 2000, technical 
problems with its sample prevent its results from being discussed here. For information on the results for the Netherlands, see OECD 
(2001). The OECD average is the average of the national averages of 27 OECD countries. Because PISA is principally an OECD study, 
the results for non-OECD countries are displayed separately from those of the OECD countries and not included in the OECD average. 



SOURCE: Organization for Economic Cooperation.and .Development, Program for International Student Assessment (PISA) 2000. 
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and males’ performance for reflecting on texts is 
higher in Iceland and Norway than in the United 
States. Among non-OECD nations, Latvia has a 
gender gap that is larger than that of the United 
States for the combined reading literacy scale and 
for the interpreting texts scale and the reflecting 
on texts scale. 

On the PISA 2000 mathematics literacy 
assessment, performance of males and females in 
the United States is similar, as it is in 16 other 
countries. Fourteen countries show higher 
performance for males than females. The size of 
the gender difference in the United States is not 
different than that of other countries. 

For most countries, including the United States, 
males and females perform similarly on the 
science literacy assessment. However, in 
Austria, Denmark, and Korea, males outperform 
females. In Latvia and New Zealand, females 
outperform males. 

PISA 2000’s findings for males and females in 
mathematics for the United States, in which males 
and females perform similarly, are generally 
consistent with the findings of other studies of 
these subjects such as NAFP and TIMSS, despite 
differences in frameworks and in some cases age 
of students assessed (Braswell et al. 2001; Mullis 
et al. 2000; Gonzales et al. 2000). NAEP, 

TIMSS, and TIMSS-R do, however, show 
differences in performance for science 
achievement of eighth-grade males and females 
that PISA 2000 does not. 



Parents’ Education 

Students with more highly educated parents tend 
to perform better on assessments, as shown for 
instance in NAEP and TIMSS (Braswell et al. 
2001; Gonzales et al. 2000). Research has 
shown that more highly educated parents are 
able to create enriched environments at home in 
which students are exposed to reading materials 
and other educational stimuli. More highly 
educated parents are also more likely to be 
involved in and show interest in their child’s 
school, to help with homework, and to have 
higher expectations for their child’s performance 



in school, all of which can influence student 
performance on assessments (Hernandez 1993). 
Parents’ education is an important student 
background factor that may be especially 
influential on assessments of everyday literacy, 
such as PISA, since the acquisition of everyday 
literacy is less dependent on school curricula 
than skills that are gained almost exclusively 
through schooling, such as knowledge of specific 
mathematical formulas, for instance. The range 
of student performance on PISA according to the 
educational level of their parents can provide 
important information about the relative 
influence of parents’ education on students’ 
literacy performance across countries. 

In the PISA 2000 background questionnaire, 
students are asked whether their parents have less 
than a high school diploma, a high school 
diploma, or a college degree. The highest level of 
education attained by either parent as reported by 
the participating 15-year-old is used in this 
analysis. Figure 16 (table A3. 20) displays average 
scores on the PISA 2000 combined reading 
literacy scale for students in each country, 
grouped according to the level of education of 
their parents. The results for mathematics and 
science literacy show a similar pattern, and are 
therefore not shown to avoid repetition. 

It can be seen from the distance between the 
average scores that parents’ education has a 
strong relationship to reading literacy. In all of 
the 29 countries with data except Norway, 
students whose parents have less than a high 
school education perform at lower levels than 
students whose parents have a high school 
degree. In all countries, students whose parents 
have less than a high school education perform 
less well than those whose parents have a college 
degree. In 13 countries, students whose parents 
have a high school degree perform less well than 
those with a college degree, but in 15 other 
countries, there is no difference in scores between 
these two groups. In Belgium, students whose 
parents have a high school diploma score higher 
than those whose parents have a college degree. 

In the United States, there is a 93 point gap in 
performance between students, whose parents 
went to college and those whose parents have 
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Figure 1 6. — Differences in combined reading literacy scores of 1 5-year-olds by parents' level of 
education, by country: 2000 
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# Data for country in one or more reporting categories are too small to report. 

NOTE: The points on each line displayed above represent the national averages for students based on the highest level of education 
attained by one or both parents: parents with less than a high school diploma, parents with a high school diploma, and parents who 
graduated college. For example, U.S. students whose parents graduated college averaged a score of 536 in reading literacy compared to 
497 for students with parents who held only a high school diploma, and 443 for students whose parents possess less than a high school 
diploma. Although the Netherlands participated in the Program for International Student Assessment (PISA) in 2000, technical problems 
with its sample prevent its results from being discussed here. For information on the results for the Netherlands, see OECD (2001). The 
OECD average is the average of the national averages of 27 OECD countries. Because PISA is principally an OECD study, the results for 
non-OECD countries are displayed separately from those of the OECD countries and not included in the OECD average. 

SOURCE: Organization for Economic Cooperation and Development, Program for International Student Assessment (PISA) 2000. 
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not completed high school on the combined 
reading literacy scale. No other country has a 
larger point difference. Norway, in contrast, 
shows a 28 point difference for the same two 
groups of students. 

In mathematics literacy, student performance is 
higher in all countries among students whose 
parents completed college than among those 
whose parents did not complete high school 
(data not shown; see table A3.21). The gap 
between these two groups’ performance is 99 
points in the United States and 20 points in 
Norway. Comparisons of student performance 
by the other levels of parents’ education yield 
more of a mixed picture. In most countries, 
including the United States, there is a difference 
in the mathematics literacy of students whose 
parents have not completed a high school degree 
and those who have. About half of the countries 
(14), including the United States, show a 
difference between the scores of students whose 
parents have completed high school and those 
whose parents have completed college. 

Science literacy follows similar patterns as 
described above (data not shown; see table 
A3 .22). In science literacy, Sweden is the only 
country in which students whose parents do not 
have a high school degree and those who do 
score similarly. In 12 of the 29 countries with 
data, including the United States, students with 
college-educated parents score higher than those 
with parents who have completed high school. In 
the United States, there is a difference of 96 
points between the performance of students 
whose parents have not completed high school, 
and those whose parents have completed college. 
The size of that difference is similar to ten other 
countries, or about a third of the countries with 
data. By contrast to the United States, in Sweden, 



the gap between the scores of those same groups 
of students is 31 points. 

The level of parents’ education, then, varies 
somewhat in its relationship to literacy across 
countries, but in the United States, it is strongly 
linked to differences in student performance in 
reading, mathematics, and science literacy. 
Students whose parents complete college show a 
clear advantage over students whose parents do 
not, and particularly over those whose parents 
have not completed high school. 

Socioeconomic Status 

The socioeconomic status of students’ parents 
consistently has been found to be among the 
strongest student background characteristics that 
influence student outcomes in the United States, 
including performance on assessments (Coleman 
et al. 1966; West, Denton, and Reaney 2000; 
Williams et al. 2000). The measure of student’s 
socioeconomic status used in PISA 2000 is based 
on the occupation of the student’s father and/or 
mother as reported by the student. 17 This in turn 
is transformed into an International 
Socioeconomic Index (ISEI) developed by 
Ganzeboom, De Graaf, and Treiman (1992), 
which allows direct comparisons between 
nations. The ISEI is keyed to the International 
Standard Classification of Occupations (ISCO). 18 

Students are assigned numbers ranging from 
about 16 to 90 on the index based on their 
parents’ occupations, so that they are arrayed on 
a continuum from low to high socioeconomic 
status, rather than placed into discrete 
categories. 19 The linkage of socioeconomic 
status to literacy scores is shown as the 
relationship between the two, rather than a 



The measure is based on the student’s report of father’s occupation, except in cases where information on father’s occupation is 
missing. In those cases, mother’s occupation was used if available. The OECD international report on PISA 2000, Knowledge and 
Skills for Life— First Restdts from the OECD Programme for International Student Assessment , bases its ISEI measure on the 
highest of the father’s or mother’s occupation. 

18 For details on the construction of this index see Ganzeboom, H., De Graaf, P., and Treiman, D. (1992) and Ganzeboom, H., and 
Treiman, D. (1996). 

19 See footnote 17. The range of ISEI scores given for the 1988 ISCO occupations listed in Ganzeboom and Treiman (1996) goes from 
16, the lowest (agricultural laborer), to 90, the highest (judge). 
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difference between group averages. Each one- 
point increase in the ISEI is associated with a 
specific increase in literacy scores, on average, 
for each country. The greater this increase in 
literacy scores in a country, the stronger the 
relationship between socioeconomic status and 
literacy scores is in that country. 

Figure 17 (page 44; table 3.23) displays the 
relationship of socioeconomic status to reading, 
mathematics, and science literacy scores for each 
participating country. For each country, the 
relationship between the ISEI and the literacy 
score is indicated in the table. For example, in 
the United States each point difference on the 
ISEI scale is associated with a 2.1 point 
difference in reading literacy on average. In 
Japan, socioeconomic background differences in 
achievement are at a minimum — a one-point 
difference in ISEI is associated with less than a 
one-point difference in literacy. By contrast 
among students in the Czech Republic, 

Germany, or Hungary, a one-point difference in 
ISEI is associated nearly a 3 point difference in 
literacy. 20 An ISEI difference of 50 points is 
roughly equivalent to the difference in 
socioeconomic status in the United States 
between a medical doctor and a motor vehicle 
mechanic, or an architect and a garbage 
collector (Ganzeboom and Treiman 1996, 
appendix A). This 50-point ISEI difference 
would translate to a score difference of 
approximately 105 points in reading — more 
than one level on the literacy scale. 

Within any one country, the relationship between 
socioeconomic status and literacy for each subject 
may vary. In the United States, the relationship of 
socioeconomic status to literacy levels is about 
the same for each subject. By contrast, in France, 
the relationship of socioeconomic status to 
literacy varies according to whether that literacy 
is reading, mathematics or science. 



In every subject area, Finland, Iceland, Japan, 
Korea, and Latvia have a smaller relationship 
between ISEI and average scores than does the 
United States. That is, average scores are less 
affected by a student’s socioeconomic 
background in these countries than they are in 
the United States. For example, the size of the 
relationship of socioeconomic background to 
reading literacy achievement is smaller in 
Finland, Iceland, Japan, Korea, and Latvia than 
in the United States. The same countries have 
smaller relationships between ISEI and average 
scores in science literacy than the United States 
does. For mathematics literacy, in addition to 
those same countries, Canada and Italy also have 
relationships of ISEI to average scores that are 
smaller than the relationship in the United States. 

There is only one case in which the size of a 
country’s relationship between ISEI and average 
scores is greater than that in the United States, in 
Germany for reading literacy. This means that 
socioeconomic background is associated with a 
larger change in average scores for reading 
literacy in Germany than in the United States. In 
every other case, the size of the relationship 
between socioeconomic background and average 
scores in the United States is similar to other 
countries. Out of 31 countries, the United States 
has a similar relationship of ISEI to average 
scores to 24 countries in reading literacy, 23 
countries for mathematics literacy, and 25 
countries for science literacy. 

In short, although a few countries show 
consistently smaller relationships of 
socioeconomic background to average scores than 
does the United States, most of the participating 
countries do not differ from the United States in 
terms of the size of the relationship between 
socioeconomic status and the literacy of their 
15-year-olds as measured by PISA 2000. 



20 Since the ISEI index has a range of 74 points {from 16 to 90), in Japan students with the lowest positions on the socioeconomic index 
would differ from those with the highest socioeconomic index positions by about 37 points. In the same way, students with the lowest 
positions on the socioeconomic index in Germany, the Czech Republic, or Hungary would differ from those with the highest by about 
192 points, or close to two standard deviations, a substantial difference. ' 
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Figure 1 7. — Relationship between parents' socioeconomic status and combined reading literacy, 
mathematics literacy, and science literacy scores, by country: 2000 




NOTE: Socioeconomic status is measured by the International Socio-Economic Index (ISEI), a measure based on the occupations of the 
student’s parent(s). The measure used in these analyses was based on the parent with the highest ISEI. Students can be placed anywhere 
from about 16 to 90 on the ISEI index. The numbers shown in the table indicate the strength of the relationship between the 
socioeconomic status and literacy in each of reading, mathematics and science. Each number is interpreted as follows: a one-unit 
difference in ISEI is associated with an n-unit difference in literacy, where ‘n’ is the number shown in the table. Thus, the larger the 
number, the greater the association between socioeconomic status and literacy. Although the Netherlands participated in the Program for 
International Student Assessment (PISA) in 2000, technical problems with its sample prevent its results from being discussed here. For 
information on the results for the Netherlands, see OECD (2001). The OECD average is the average of the national averages of 27 
OECD countries. Because PISA is principally an OECD study, the results for non-OECD countries are displayed separately from those of 
the OECD countries and not included in the OECD average. 

SOURCE: Organization for Economic Cooperation and Development, Program for International Student Assessment (PISA) 2000. 
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Parents’ Nrtionrl Origin 

Children who are foreign-born or who are 
children of foreign-born parents face challenges 
during the process of acculturating and adjusting 
to a new country, and their achievement in 
school and general literacy may suffer 
accordingly (Lollock 2001). Other OECD 
countries as well as the United States are home to 
large populations of children whose parents are 
foreign-born, and it has become an important 
goal in these countries to prevent the social 
exclusion of these children from educational and 
economic opportunity. In the United States, 14 
percent of students have both parents born 
outside the United States. 21 Australia, Canada, 
Latvia, Liechtenstein, Luxembourg, New 
Zealand, and Switzerland all have 20 percent or 
more students with two foreign-born parents. 

The Czech Republic, Finland, Italy, Iceland, 
Japan, Poland, and Brazil, in contrast, have 
1 percent or less of their students with two 
foreign-born parents (table A3. 24). 

Figure 18 (page 46) presents the reading literacy 
of students in each country by whether their 
parents were born in that country. The categories 
are: both parents are born in the country; one 
parent is native-born and one parent is foreign- 
born; and both parents are foreign-born. 

In the United States and 18 of the 26 other 
countries with data, students with two foreign- 
born parents score lower on the combined 
reading literacy scale than students with two 
native-born parents. For example, in Belgium, 
the difference between groups is almost 109 
points, while in the United States it is a 40 point 
gap. Belgium, Germany, Luxembourg, and 
Switzerland have significantly larger differences 
between these two groups than the United States. 
Likewise, in all but six countries, students with 
two foreign-born parents score lower than those 
with one native-born and one foreign-born 
parent. The difference in performance between 



these two groups in the United States is not 
significant. In most countries (22 of the 29 
countries with data), including the United States, 
there is no difference between the reading literacy 
achievement of students with two native-born 
parents and those with one foreign-born and one 
native-born parent. 

In mathematics literacy, fewer countries have a 
gap in performance between students with 
foreign-born parents and the other two groups of 
students (data not shown; see table A3. 25). In 15 
of the 26 countries with data, or about half of the 
PISA 2000 participating countries, including the 
United States, the difference between students 
with two foreign-born parents and students with 
two native-born parents is significant. For the 
U.S., this gap is about 40 points. There are no 
differences in the United States between students 
with two foreign-born parents and those with one 
native-born and one foreign-born parent, or 
between students with one native-born and one 
foreign-born parent and students with two native- 
born parents. These results suggest that the 
acquisition of mathematics skills is less influenced 
by parents’ place of birth than is the acquisition 
of reading skills. 

Science literacy varies less by parental nativity 
than reading literacy, but more than mathematics 
literacy (data not shown; see table A3. 26). In 17 
of the 26 countries with data, but not the United 
States, students with two foreign-born parents 
perform at lower levels in science literacy than 
students with two native-born parents. Fifteen- 
year-olds with one native and one foreign-born 
parent also outperform students with two 
foreign-born parents in 17 countries. In only four 
countries do students with one foreign-born and 
one native-born parent score differently than 
those with two native-born parents. In the 
United States, there is no difference in science 
literacy by parental nativity. 



1 Percentages are for the reading literacy part of the assessment; there are small differences in the percentages reported for reading 
literacy, mathematics literacy, and science literacy because of slight differences in the numbers of students taking each part of the 
assessment. For example, in the United States percentages reported for students with both parents bom outside the country are 14 
percent for reading literacy and science literacy and 13 percent for mathematics literacy. 
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Figure 1 8. — Differences in combined reading literacy scores of 1 5-year-olds by parents' national origin, 
by country: 2000 
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of the test country in the test country in the test country 



# Data for country in one or more reporting categories are too small to report. 

NOTE: The points on each line displayed above represent the national averages for students based on their parents’ national origin: 
students with both parents born in the test country, students with one parent bom in the test country, and students with parents both 
born outside of the test country. For example, U.S. students whose parents were born in the United States averaged a score of 512 in 
reading literacy compared to 494 for students with one parent born in the United States, and 472 for students whose parents were born 
outside of the United States. Data for the Republic of Korea are not available. Although the Netherlands participated in the Program 
for International Student Assessment (PISA) in 2000, technical problems with its sample prevent its results from being discussed here. 
For information on the results for the Netherlands, see OECD (2001). The OECD average is the average of the national averages of 26 
OECD countries. Because PISA is principally an OECD study, the results for non-OECD countries are displayed separately from those 
of the OECD countries and not included in the OECD average. 

SOURCE: Organization for Economic Cooperation and Development, Program for International Student Assessment (PISA) 2000. 
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Language Spoken at Home 

Over the last two decades, the percentage of 
students in U.S. schools speaking a language 
other than English at home has more than 
doubled to 17 percent in 1999 (Federal 
Interagency Forum on Child and Family 
Statistics 2001). These students may face greater 
challenges progressing in school and in the labor 
market, once they leave school (Federal 
Interagency Forum on Child and Family 
Statistics 2001). The United States is not unique 
in this respect. Other OECD nations are also 
educating students whose first language is not 
the language of instruction in the country and/or 
who speak another language at home. 

Students who speak a language at home other 
than the language in which the assessment is 
given may be more likely to lack a facility in the 
language of the assessment, which could affect 
performance in reading, mathematics, and 
science literacy. In PISA 2000, students were 
asked what language they speak at home most 
of the time — the language of the assessment, 
English, or another language. 22 Two countries, 
Hungary and Korea, do not report information 
for this question. One other, Japan, has 
numbers of students reporting speaking a 
language other than Japanese that are too small 
to reliably estimate scores for that group. The 
percentage" of students who respond that they 
speak the test language most of the time at 
home ranges from 69 percent in Canada to 99 
percent in Brazil. In the United States, 89 
percent of students report that they speak the 
language of the assessment (English) at home 
most of the time (table A3.27). 23 

Dividing students into two categories — those who 
speak the test language at home most of the time, 
and those who do not — allows a comparison of 
their performance. Achievement differences 



between these two groups are shown in figure 19 
(tables A3.27, A3. 28, and A3. 29) for reading, 
mathematics, and science literacy in each of the 
participating nations. 

In most countries, there is a gap in reading 
achievement for students who speak a language 
other than the assessment language at home. 
Except for Brazil, the Czech Republic, Ireland 
and Spain, the reading achievement of students 
who speak the test language at home is higher 
than that of students not speaking this language 
at home. The size of the difference is similar to 
that of the United States for most countries; 
however, Australia, Belgium, and Canada show a 
smaller achievement difference than the 
difference between these groups of students in 
the United States. 

The second graph in figure 19 (page 48) displays 
the differences in performance for mathematics 
literacy by the language spoken in the home. 
Again, students who speak the language of the 
test show an advantage in most countries. 
Nineteen of the 28 nations that report results 
show an advantage for test-language speakers in 
mathematics literacy, including the United States. 
In two countries, Canada and Spain, the 
difference favors students who do not speak the 
language of the test at home “most of the time.” 
Six countries (Australia, Belgium, Canada, 
Finland, Italy, and Spain) have smaller 
achievement differences in mathematics literacy 
between students who speak the test language 
and those who do not than does the United 
States. The U.S. difference between test and non- 
test language speakers is larger than the OECD 
average difference for mathematics literacy. 

The third graph in figure 19 displays the 
differences between the two language-defined 
groups in science literacy achievement. As in the 
case of reading and mathematics, the findings 
point to an advantage in speaking the language 



22 Other PISA countries included response options for students to choose an official national language or a national dialect. In this 
report, for these countries, all these responses are grouped as languages other than the test language. 

23 See footnote 21. 
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Figure 19. — Differences in combined reading literacy, mathematics literacy and science literacy scores of 
1 5-year-olds by language spoken at home, by country: 2000 
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NOTE: Each bar above represents the average score difference on combined reading, mathematics or science literacy between students 
who speak the test language at home and those who speak another language. Some of these differences are statistically significant and 
indicated by darker bars. For example, in the United States, students who speak English at home outperform students who speak another 
language at home in all three subject areas. These score differences are 76 points in combined reading literacy, 73 points in mathematics 
literacy and 68 points in science literacy. Data for Hungary and the Republic of Korea are not available. Data for Japan on students who 
speak other languages at home are too small to report, and therefore, data for Japan are not presented. Average score difference is 
calculated by subtracting the average score for those who speak other languages at home from average scores for those who speak the test 
language at home in each country. Detail may not sum to totals due to rounding. Although the Netherlands participated in the Program 
for International Student Assessment (PISA) in 2000, technical problems with its sample prevent its results from being discussed here. For 
information on the results for the Netherlands, see OECD (2001). The OECD average is the average of the national averages of 25 
OECD countries. Because PISA is principally an OECD study, the results for non-OECD countries are displayed separately from those of 
the OECD countries and not included in the OECD average. 



SOURCE: Organization for Economic Cooperation and Development, Program for International Student Assessment (PISA) 2000. 
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of the test. In 21 of the 28 nations that report 
results for this question, including the United 
States, test-language speakers perform better 
than speakers of other languages. The remaining 
countries show no differences in performance 
for science literacy. Belgium has a smaller 
difference between these groups than the United 
States does. 

Overall, while there is some variability among 
countries in this respect, in the United States and 
in most other countries, the language that 
students speak at home is linked to their 
performance on PISA 2000’s reading, 
mathematics, and science literacy assessments. 

Race and Hispanic Origin 

Differences in performance by race and Hispanic 
origin on the National Assessment of Educational 
Progress (NAEP) have been documented for more 
than two decades (Campbell et al. 1996). White 
students tend to score higher on assessments than 
Blacks and Hispanics. 24 This same pattern 
emerges from the U.S. components of 
international assessments as well (Binkley and 
Williams 1996; Gonzales et al. 2000). A 
substantial amount of research addresses these 
between-group differences, focusing on social and 
economic background, differences in quality of 
social and educational environments, and 
inequality of opportunity brought on by direct 
and indirect discriminatory practices (Wilson 
1996; Hedges and Nowell 1999). 

Racial and ethnic groups vary between countries, 
so it is not possible to compare their performance 
across countries on international assessments. 
Thus this section refers only to findings for the 
United States. Students’ race and Hispanic origin 
is obtained through student responses to a 
two-part question. Students are asked first 
whether they are Hispanic, and then asked 
whether they are members of the following racial 
groups — American Indian/Alaska Native, Asian, 



Black or African American, Native Hawaiian or 
Pacific Islander, or White. Multiple responses are 
allowed so students can be identified as 
multiracial. Students identifying themselves as 
Hispanic and also Black or White are included in 
the Hispanic group. Because of small numbers of 
students, all identifications other than the three 
major groups of White, Black, and Hispanic are 
grouped as “other.” This includes multiracial 
students. Comparisons among these four groups 
are presented for reading, mathematics, and 
science literacy using these categories. 

Figure 20 (page 50; table A3. 30) displays the 
average performance levels of U.S. 15-year-olds in 
reading, mathematics, and science literacy for 
each of the four identified racial and ethnic 
groups. The chart presents a comparison of each 
group by each other group. Average scores for 
each group appear to the right of the chart. The 
letters in the chart indicate (by the first letter of 
the word for the group) differences in 
achievement between groups being compared. In 
reading literacy, for example, Whites outperform 
both Hispanics and Blacks. 

Figure 20 makes it clear that the pattern of 
between-group differences is identical across the 
three literacy areas. In reading, mathematics, and 
science, the average literacy scores for Whites and 
other students are higher than for Hispanic and 
Black students. This pattern of performance on 
PISA 2000 by race and Hispanic origin is similar 
to that on NAEP and other assessments 
(Campbell et al. 1996). 



24 Whites refers to non-Hispanic Whites and Blacks to non-Hispanic Blacks. 
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Figure 20. — Comparisons of reading, mathematics and science literacy average scores of U.S. 1 5-year- 
olds, by race/ethnicity: 2000 
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White students outperformed compared racial/ethnic group 
Other students outperformed compared racial/ethnic group 
No significant score differences between compared groups 




*The “other” group comprises students identifying themselves as American Indian/Alaska Native, Asian, Native Hawaiian/Pacific 
Islander, or multiracial since numbers of these students are too small to report by individual categories. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Program for International Student Assessment (PISA), 
2000. 



Summary 

Just as the chapters on reading, mathematics, and 
science literacy began with a simple measure of 
average performance for a country and then 
presented additional information on how 
performance varies within and between countries, 
this chapter shows further variations in 
performance based on background characteristics 



of 15-year-olds. Most comparisons are both 
within and across nations, although comparisons 
by race and Hispanic origin are for the United 
States only. The next chapter will show some of 
the initial steps PISA has taken to measure general 
outcomes of learning such as attitudes toward 
learning and learning strategies, and will continue 
some of the discussion begun in this chapter about 
gender differences in learning outcomes. 
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Cross-curricular 
Competencies and the 
Future of PISA 




o Thirty percent of U.S. 15-year-olds agree or strongly agree that reading is a 
favorite hobby, a lower percentage than the OECD average. Percentages range 
from 62 percent of students agreeing that reading is a favorite hobby in 
Mexico to 24 percent in Norway (figure 21; table A3 .31). 

o In every country, females agree more frequently than males that reading 
is a favorite hobby. Thirty-seven percent of females in the United States agree 
that reading is a favorite hobby, compared to 22 percent of males (figure 22; 
table A3. 31). 

o About half of U.S. 15-year-olds report trying to memorize as much as possible 
often or always when studying. The U.S. percentage in this case is higher than 
the OECD average, suggesting that a greater proportion of U.S. students often 
use memorization as a learning strategy than the average proportion of OECD 
country students (figure 23; table A3. 32). 

o The percentages of students who respond that they often or always try to 
relate new material to things they have already learned range from 15 percent 
in Italy to 90 percent in Hungary. Fifty-nine percent of U.S. students report 
using this strategy frequently, a higher percentage than the OECD average 
(figure 25; table A3. 33). 
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One of PlSA’s main objectives is to measure 
student performance on general or nonacademic 
learning outcomes in addition to outcomes for 
reading, mathematics, and science literacy. These 
“cross-curricular competencies,” or CCCs, will 
have a growing importance in PISA as it develops 
over time. The measurement of these kinds of 
competencies is part of PlSA’s mission to measure 
a variety of important knowledge and skills 
needed in adult life. 

In 2003, for example, PISA will assess students 5 
abilities to solve problems. Plans for PISA 2006 
include the development of an assessment of 
students 5 abilities to use information and 
communications technologies. 

As a first step toward the measurement of cross- 
curricular competencies, in PISA 2000, student 
questionnaire items sought information in two 
major learning areas, student attitudes toward 
learning and learning strategies. These data can 
be viewed both as an input for student learning as 
well as outcomes of students 5 school and life 
experiences up until the age of 15. For example, 
are certain kinds of learning strategies associated 
with higher performance in reading, mathematics, 
or science literacy? By the age of. 15, what kind 
of learning strategies and what kinds of attitudes 
toward learning do students have? This chapter 
will present information on student reports for 
questions related to these topics, including some 
discussion of gender differences. 25 

Attitudes Towards Reading 

Proficiency scores for reading literacy describe 
how well 15-year-olds can apply a variety of 
reading processes in different kinds of situations. 



They do not address, however, students’ attitudes 
toward reading or how likely students are to 
apply their reading literacy skills. To develop a 
fuller picture of what 15-year-old students are 
like as learners, PISA 2000 also included a 
number of questions for students about their 
attitudes toward reading and math and time 
spent reading. Only information related to 
reading is discussed here. 26 

One indication of how students view reading and 
how important it is to them is the extent to which 
they read in their free time (figure 21). PISA 
2000 asked students to provide information on 
this issue by rating the statement “Reading is one 
of my favorite hobbies.” 27 

Thirty percent of U.S. students agree or strongly 
agree that reading is a favorite hobby, a lower 
percentage than the OECD average. Percentages 
range from 62 percent of students agreeing that 
reading is a favorite hobby in Mexico to 24 
percent in Norway. 

In the United States and in every other country, 
females agree more frequently than males that 
reading is a favorite hobby (figure 22, page 54; 
table A3.31). Thirty-seven percent of females in 
the United States agree that reading is a favorite 
hobby, compared to 22 percent of males. In a 
number of countries (Austria, Canada, Czech 
Republic, Finland, Germany, Ireland, Portugal, 
and Switzerland), the difference between males 
and females is greater than in the United States. 

By the age of 15, greater proportions of young 
people in other OECD countries cite reading as a 
favorite pastime than in the United States. Why 
do greater percentages of non-U.S. students say 
this? Also, why do more females consider 



5 Analysis of attitudes toward learning and learning strategies was limited to differences by gender for this initial report on PISA 2000 
because of the restricted nature of the measures of the cross-curricular competencies discussed in this report. Differences for other 
population subgroups (such as racial or ethnic groups) could be analyzed in future reports or cycles of PISA. 

26 PISA 2000 created indices of interest in reading and mathematics from a series of student questions. Only an illustrative question is 
discussed here. For further information about these indices, see the PISA international report Knowledge and Skills for Life — First 
Results from the OECD Programme for International Student Assessment (OECD 2001). 

27 In interpreting these results, readers should keep' in mind that these are students’ own descriptions of their behaviors, and that 
differences in responses may be in part attributable to cultural differences or social norms in participating countries. 
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Figure 21 . — Percentage of 1 5-year-olds who agree or strongly agree that reading is a favorite hobby, 
by country; 2000 
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NOTE: Although the Netherlands participated in the Program for International Student Assessment (PISA) in 2000, technical problems 
with its sample prevent its results from being discussed here. For information on the results for the Netherlands, see OECD (2001). The 
OECD average is the average of the national averages of 27 OECD countries. Because PISA is principally an OECD study, the results for 
non -OECD countries are displayed separately from those of the OECD countries and not included in the OECD average. 

SOURCE: Organization for Economic Cooperation and Development, Program for International Student Assessment (PISA) 2000. 



reading a favorite hobby both in the United States 
and elsewhere? These data raise more questions 
than PISA 2000 alone can answer, but further 
study of this issue could hold lessons for schools, 
parents, and policymakers about reading. 



Learning Strategies 

The idea of lifelong learning has become an 
important part of the educational landscape in 
the past few years. The notion is that formal 
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Figure 22. — Percentage differences by gender of 1 5-year-olds who agree or strongly agree that reading 
is a favorite hobby, by country: 2000 



Country Males Females 



Australia 
Austria 
Belgium 
Canada 
Czech Republic 
Denmark 
Finland 
France 
Germany 
Greece 
Hungary 
Iceland 
Ireland 
Italy 
Japan 

Korea, Republic of 
Luxembourg 
Mexico 
New Zealand 
Norway 
Poland 
Portugal 
Spain 
Sweden 
Switzerland 
United Kingdom 
United States 

OECD average 

Non-OECD countries 

Brazil 
Latvia 
Liechtenstein 
Russian Federation 
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NOTE: Each bar above represents the difference in percentages of males and females in each country who agree or strongly agree that 
reading is a favorite hobby. These differences are statistically significant in every country and indicated by darker bars. For instance, 
in the United States, 15 percent more females than males agree or strongly agree that reading is a favorite hobby, a difference that is 
statistically significant. Average percentage difference is calculated by subtracting percentage of males who agree from percentage of 
females who agree. Detail may not sum to totals due to rounding. Although the Netherlands participated in the Program for 
International Student Assessment (PISA) in 2000, technical problems with its sample prevent its results from being discussed here. For 
information on the results for the Netherlands, see OECD (2001). The OECD average is the average of the national averages of 27 
OECD countries. Because PISA is principally an OECD study, the results for non-OECD countries are displayed separately from those 
of the OECD countries and not included in the OECD average. 

SOURCE: Organization for Economic Cooperation and Development, Program for International Student Assessment (PISA) 2000. 
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schooling cannot provide all of the knowledge 
and skills that people will need throughout their 
lives, and that people need to continue to learn 
(both formally and informally) for their own 
individual social and economic well-being and 
for the well-being of their societies. 
Acknowledging that students cannot learn 
everything they need for success in life in school, 
PISA recognizes that students must at least 
develop the prerequisites for successful learning. 
Research shows that these prerequisites are 
cognitive and motivational in nature and that 
several dimensions (i.e., beliefs, attitudes) are 
related to self-regulated learning (Baumert et al., 
1998). The PISA instrument on self-regulated 
learning focused on three of these dimensions — 
learning strategies, motivation, and self-concept. 
The previous section touched upon motivation; 
this section focuses on learning strategies. 

PISA 2000 provides initial information on the 
different strategies for learning that students 
report using. This information can provide some 
insight into how schools and societies have shaped 
young people’s approaches to learning and how 
students see themselves as learners. Young people 
may take these approaches to learning with them 
into adulthood, and awareness of what learning 
strategies they use may help them to be successful 
learners in the future. 

To collect this information, PISA 2000 offered 
participating countries a series of optional 
questions for students on their learning strategies. 
Questions related to learning strategies focused 
on several areas: control of learning, use of 
memorization and elaboration strategies, and the 
use of competitive or cooperative strategies for 
success. Only information related to the use of 
memorization and elaboration strategies is 
discussed here. 28 



To gauge students’ use of memorization strategies 
when studying, PISA 2000 asked students to 
reply to the question “I memorize as much as 
possible” by choosing one of four possible 
responses: never, sometimes, often, or always. 29 

About half (49 percent) of U.S. 15-year-olds 
report trying to memorize as much as possible 
often or always when studying (figure 23, page 
56). The U.S. percentage in this case is higher 
than the OECD average, suggesting that a greater 
proportion of U.S. students often use 
memorization as a learning strategy than the 
average proportion of OECD country students. 

In the United States, about the same percentages 
of males and females (50 percent of females and 
48 percent of males) say they often or always try 
to memorize as much as possible when studying 
(figure 24, page 57; table A3. 32). Fourteen other 
countries (of the 25 countries reporting data) also 
have similar percentages of males and females 
reporting this strategy, but the remaining 10 other 
countries that report information on the use of 
memorization all have more males than females 
who say they use this strategy. 

As a contrast to memorizing while studying, PISA 
2000 also asked students to reply to the question 
“I try to relate new material to things I have 
already learned in other subjects,” using the same 
response categories noted above for 
memorization: never, sometimes, often, and 
always. PISA 2000 considers this type of question 
as an indication of use of “elaboration” strategies. 

Percentages of students who respond that they 
often or always try to relate new material to 
things they have already learned range from 15 
percent in Italy to 90 percent in Hungary. Fifty- 
nine percent of U.S. students (figure 25, page 58) 



28 Elaboration strategies refer to students’ relating information they are trying to learn to things they already know or have learned, or 
trying to relate new things to real world situations. PISA 2000 created indices for memorization and elaboration from a series of 
student questions. Only responses to two illustrative questions are discussed here. For further information on learning strategies or 
other cross-curricular competencies, see the PISA international report Knotvledge and Skills for Life —First Results from the OECD 
Programme for International Student Assessment (OECD 2001). An in-depth report on these topics is planned to be published in 
2002. 

29 Questions on learning strategies were optional, and not all countries report data for this question. Canada, France, Greece, Japan, 
Poland, and Spain do not report results. 
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Figure 23. — Percentage of 1 5-year-olds who report memorizing often or always when studying, by 
country: 2000 



Country 



United Kingdom 
Hungary 
Mexico 
United States 

New Zealand 
Australia 
Czech Republic 
Germany 
Switzerland 
Italy 
Austria 
Ireland 
Portugal 
Sweden 
Finland 
Denmark 
Iceland 
Norway 
Korea, Republic of 
Belgium 
Luxembourg 

OECD average 

Non-OECD countries 

Brazil 
Latvia 
Liechtenstein 
Russian Federation 





S 



Percent 



Percentage is significantly higher than U.S. percentage 
Percentage is not significantly different from U.S. percentage 
Percentage is significantly lower than U.S. percentage 



NOTE: Although the Netherlands participated in the Program for International Student Assessment (PISA) in 2000, technical problems 
with its sample prevent its results from being discussed here. For information on the results for the Netherlands, see OECD (2001). The 
OECD average is the average of the national averages of 21 OECD countries. The other participating OECD countries did not collect 
data for this question. Because PISA is principally an OECD study, the results for non-OECD countries are displayed separately from 
those of the OECD countries and not included in the OECD average. 

SOURCE: Organization for Economic Cooperation and Development, Program for International Student Assessment (PISA) 2000. 
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Figure 24. — Percentage differences by gender of 1 5-year-olds who report memorizing often or always 
when studying, by country; 2000 




Country Males Females 



Australia 
Austria 
Belgium 
Czech Republic 
Denmark 
Finland 
Germany 
Hungary 
Iceland 
Ireland 
Italy 

Korea, Republic of 
Luxembourg 
Mexico 
New Zealand 
Norway 
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Switzerland 
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United States 

OECD average 
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Percentage difference 



Percentage difference is statistically significant 
Percentage difference is not statistically significant 




NOTE: Each bar above represents the difference in percentages of males and females in each country who say the use memorization 
often or always. Some of these differences are statistically significant and indicated by darker bars. In the United States, 2 percent 
more females than males state they use memorization often or always, a difference that is not statistically significant and indicated by a 
lighter bar. Average percentage difference is calculated by subtracting percentage of males who agree from percentage of females who 
agree. Detail may not sum to totals due to rounding. Although the Netherlands participated in the Program for International Student 
Assessment (PISA) in 2000, technical problems with its sample prevent its results from being discussed here. For information on the 
results for the Netherlands, see OECD (2001). The OECD average is the average of the national averages of 21 OECD countries. 

The other participating OECD countries did not collect data for mis question. Because PISA is principally an OECD study, the results 
for non-OECD countries are displayed separately from those of the OECD countries and not included in the OECD average. 

SOURCE: Organization for Economic Cooperation and Development, Program for International Student Assessment (PISA) 2000. 
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Figure 25. — Percentage of 1 5-year-olds who report using an elaboration strategy often or always when 
studying, by country: 2000 



Country 



Hungary 
United Kingdom 
Australia 
New Zealand 
Ireland 
Sweden 
United States 
Denmark 
Iceland 
Belgium 
Korea, Republic of 
Mexico 
Switzerland 
Portugal 
Czech Republic 
Germany 
Luxembourg 
Finland 
Austria 
Norway 
Italy 

OECD average 

Non-OECD countries 

Brazi 
Latvia 
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Percent 



Percentage is significantly higher than U.S. percentage 
Percentage is not significantly different from U.S. percentage 
Percentage is significantly lower than U.S. percentage 



NOTE: Elaboration refers to students’ relating information they are attempting to learn to things they already know or have learned. 
Although the Netherlands participated in the Program for International Student Assessment (PISA) in 2000, technical problems with its 
sample prevent its results from being discussed here. For information on the results for the Netherlands, see OECD (2001). The OECD 
average is the average of the national averages of 21 OECD countries. The other participating OECD countries did not collect data for 
this question. Because PISA is principally an OECD study, the results for non-OECD countries are displayed separately from those of the 
OECD countries and not included in the OECD average. 

SOURCE: Organization for Economic Cooperation and Development, Program for International Student Assessment (PISA) 2000. 
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report using this strategy frequently, a higher 
percentage than the OECD average. 

Within the United States, a higher proportion of 
females (61 percent) than males (57 percent) say 
they frequently try to relate new material to 
things they have already learned. This difference 
between females and males is also evident in 
eight other countries (figure 26, page 60; table 
A3.33). Six countries show the opposite 
pattern, in which greater percentages of males 
than females report frequently trying to relate 
new material to things they have already 
learned, and the remaining countries show no 
differences between males and females. 

Memorization and elaboration strategies are, of 
course, not mutually exclusive learning 
approaches, and about half of U.S. 15-year-olds 
report using each of these strategies frequently. In 
both cases, the U.S. percentage is higher than the 
OECD average percentage. For the U.S., greater 
percentages of females than males report using 
each strategy frequently. Again, this preliminary 
information from PISA 2000 raises more 
questions than it can answer. Why do greater 
proportions of females in the United States report 
using elaboration as a strategy? Why do higher 
percentages of U.S. students report trying to 
memorize as much as possible than in other 
OECD countries? What can this tell us about 
how 15-year-olds think about learning? How 
might these learning strategies affect students as 
they continue on in their lives? 



The Future of PISA 

As noted above, as PISA develops over time, it is 
planned that general outcomes of learning or 
cross-curricular competencies will come to play a 
greater role and that the measures of reading 
literacy, mathematics literacy, and science literacy 
will be continually refined and improved. 

This report provides an initial look at results 
from PISA 2000 from a U.S. perspective. As 
mentioned, the Organization for Economic 
Cooperation and Development (OECD) has also 
published a report describing results from PISA 
2000. Additionally, over the next 2 years, the 
OECD will publish a series of additional reports 
addressing specific themes or questions from 
PISA 2000 in greater detail. These reports will 
include an in-depth look at reading literacy, and 
will also deal in greater depth with issues such as 
gender and socioeconomic status. Data from 
PISA 2000 will also be made available to the 
public and to researchers. As with any 
international study, PISA’s value is largely 
determined by the countries participating in it. 
Hopefully, the information contained in this 
report will be interesting and useful for U.S. 
policymakers, researchers, and the public. 
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Figure 26. — Percentage differences by gender of 1 5-year-olds who report using an elaboration strategy often or 
always when studying, by country: 2000 
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NOTE: Elaboration refers to students’ relating information they are attempting to learn to things they already know or have learned. 
Each bar above represents the difference in percentages of males and females in each country who say they use an elaboration strategy 
often or always. Some of these differences are statistically significant and indicated by darker bars. For instance, in the United States, 4 
percent more females than males state they use an elaboration strategy often or always, a difference that is statistically significant. Average 
percentage difference is calculated by subtracting percentage of males who agree from percentage of females who agree. Detail may not 
sum to totals due to rounding. Although the Netherlands participated in the Program for International Student Assessment (PISA) in 
2000, technical problems with its sample prevent its results from being discussed here. For information on the results for the Netherlands, 
see OECD (2001). The OECD average is the average of the national averages of 21 OECD countries. The other participating OECD 
countries did not collect data for this question. Because PISA is principally an OECD study, the results for non-OECD countries are 
displayed separately from those of the OECD countries and not included in the OECD average. 

SOURCE: Organization for Economic Cooperation and Development, Program for International Student Assessment (PISA) 2000. 
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Sampling Procedures and 
Response Rates 

PISA 2000 developed quality standards, 
procedures, instruments, and verification 
mechanisms to ensure that national samples 
yielded comparable data. Experts from the PISA 
Consortium monitored the sample selection 
process in each participating country. 1 PISA’s data 
quality standards required minimum participation 
rates for educational institutions as well as for 
students. These standards were established to 
minimize the potential for response biases. 

A minimum response rate target of 85 percent 
was required for initially selected educational 
institutions. In instances in which the initial 
response rate of educational institutions was 
between 65 and 85 percent, an acceptable school 
response rate could still be achieved through the 
use of replacement schools. 

PISA 2000 also required a minimum 
participation rate of 80 percent of students 
within participating educational institutions 
(sampled and replacement). A student was 
considered to be a participant only if he or she 
participated in the first testing session. The 
minimum participation rate had to be met at the 
national level, not necessarily for each 
participating educational institution. 

In the United States, the public and private 
schools selected for PISA constituted a nationally 
representative sample of all schools in the country 
enrolling 15-year-olds. A three-stage sampling 
design was implemented: the first stage was a 
sample of primary sampling units (geographic 
areas referred to as PSUs); the second stage was a 
sample of schools within PSUs; and the third 
stage was a sample of students from the set of all 



students enrolled in the school who were born in 
the calendar year 1984. 

In the first stage of sampling, 52 PSUs were 
selected. During the second stage, a total sample 
of 220 schools was selected from within the 
sampled PSUs. International requirements 
specified that a minimum of 150 schools be 
selected. This number was increased to 220 in 
the United States to offset school nonresponse, 
design effects from the three-stage design, and 
design effects from oversampling of high 
minority schools. The selected schools were 
located in 33 different U.S. states. 

As a supplement to the PISA school sample 
described above, replacement schools were 
selected from the unsampled schools on the 
sampling frame. Each school in the original 
sample was assigned up to two replacement 
schools selected from the set of “neighboring” 
schools on the sampling frame. As the sampling 
frame is ordered by school characteristics, these 
neighboring frame schools have similar 
characteristics to the sampled school, and their 
addition to the sample can reduce the 
nonresponse bias incurred from the lack of 
cooperation of the sampled school. 

Ten of the 220 schools in the original sample 
were ineligible because they did not have any 
students born in 1984, and a further 82 schools 
refused to participate, leaving 128 schools before 
replacement. Thirty-two replacement schools 
agreed to participate with the result that 160 
schools in total agreed to participate in the study. 
Following data collection, decisions by the 
international Technical Advisory Group (made 
up of technical advisors from the PISA 
Consortium) reduced the number of 
“participating” schools based on the student 



1 The PISA Project Consortium consists of the Australian Council for Educational Research (ACER), the Netherlands National Institute 
for Educational Measurement (CITO), Educational Testing Service (ETS, USA), National Institute for Educational Research (NIER, 
Japan), and Westat (USA). ACER coordinates the Consortium, under contract to the OECD. In the United States, Westat carried out 
the PISA assessment. 
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response rates within schools. Schools with more 
than 50 percent student participation were 
classified as “responding schools.” Schools in 
which 25 to 50 percent of the sampled students 
participated were classified as “partially 
responding.” Schools with less than 25 percent 
student participation were treated as 
“nonresponding,” and data from these schools 
were deleted from the database. In the United 
States the number of (original/replacement) 
schools falling into these categories was as 
follows: responding, (116/29); partially 
responding, (7/1); and nonresponding, (5/2). For 
the purposes of calculating school response 
rates only the 145 responding schools (116 
originals plus 29 replacements) were counted. 

On this basis the school response rate before 
replacement was 56 percent, and after 
replacement became 70 percent. 

In total some 4,752 students were sampled from 
the 145 responding schools. Eligible students 
were defined as those born in 1984 and in each 
school a random sample of up to 35 of these 
eligible students was selected. Some 221 of these 
students were subsequently classified as ineligible 
and/or were withdrawn. Exclusion decisions by 
schools resulted in a further 211 students being 
excluded from the assessment. The sampling plan 
provided for sampling from all 15-year-old 
students within a school. Some of the selected 
students could have an Individualized Education 
Plan (IEP) or be identified by the school as 
limited English proficient (LEP). School staff who 
were knowledgeable about the school’s IEP/LEP 
students reviewed the list of selected students to 
determine whether any of them had an IEP or 
were identified as LEP. School staff identified 
those students that they felt were unable to 
meaningfully participate in the assessment. Not 
all IEP/LEP students were excluded — the 
following guidelines were used to determine 
which students would participate: 

o Functionally disabled students. These are 
students who are permanently physically 
disabled in such a way that they cannot 
perform in the testing situation. Functionally 



disabled students who can respond should be 
included in the testing. Any sampled student 
who is temporarily disabled such that s/he 
cannot participate in the assessment will be 
considered absent from the assessment. 

o Students with mental or emotional 
disabilities. These are students who are 
considered in the professional opinion of the 
school principal or by other qualified staff 
members to be educable mentally retarded or 
who have been psychologically tested as 
such. This includes students who are 
emotionally or mentally unable to follow 
even the general instructions of the test. 
Students should not be excluded solely 
because of a poor academic performance or 
normal disciplinary problems. 

o Students with limited proficiency in the test 
language (English). These are students who 
are actually unable to read or speak the 
language of the test (English) and would be 
unable to overcome the language barrier in 
the test situation. Typically, a student who 
has received less than 1 year of instruction in 
the language of the test should be excluded; 
all others should be included. 

The students excluded followed the guideline 
categories as follows: 39 percent were students 
with mental or emotional disabilities, 33 percent 
had limited English language proficiency, 24 
percent were functionally disabled, and 4 percent 
were excluded for other reasons, including being 
home-schooled and participating temporarily in a 
drug rehabilitation program. In line with the 
internationally specified procedures, no special 
attempts were made to accommodate students 
with physical disabilities over and above those 
provided by the school itself. 

The result of this attrition due to ineligibility, 
withdrawal, or exclusion was that 4,320 
students were eligible to take the assessment. Of 
these, 620 students failed to take the assessment 
due to absence and/or parent/student refusals. In 
total then 3,700 students from the 145 
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responding schools were assessed. The weighted 
number of students assessed, expressed as a 
percentage of the weighted number of eligible 
students, gave the student response rate of 85 
percent, a rate which exceeds the PISA 
international standard of 80 percent. In addition, 
146 students in the partially responding schools 
took the assessment giving a total of 3,846 
students taking the PISA assessment in the 
United States. All 3,846 students are included in 
the international database. 

While the student response rate exceeds both 
NCES and PISA standards, the school response 
rate of 56 percent before replacement fails to 
meet these standards. In the case of PISA a rate 
of 65 percent was required. The United Kingdom 
and the Netherlands also fell below the PISA 
standard for response rates. Each nation 
undertook analyses designed to examine the 
extent of bias, if any, introduced by this level of 
nonresponse. Since assessment data are not 
available for the nonresponding schools, the 
analysis of the PISA data for the United States 
compared participants and nonparticipants in the 
original and original plus replacement samples 
using logistic regression to predict participation. 
The predictors in question were sampling frame 
school variables with a history of association 
with student achievement in various national 
assessments — region, metropolitan/nonmetro- 
politan, public/private, type of school, percentage 
minority, percent eligible for free lunch, 
estimated number of 15-year-olds, and school 
grade-span. These analyses indicate that there are 
differences between responding and 
nonresponding schools in some of these respects. 
Region, metropolitan/nonmetropolitan status, 
percentage minority, and percentage eligible for 
free lunch were found to be significant predictors 
of school nonresponse. In addition, there was a 
nonlinear relationship with minority (Black and 
Hispanic) enrollment — schools with relatively 
high, and relatively low, minority enrollment 
were considerably more likely to participate than 
those with intermediate levels of minority 
enrollment. While the implications of these 
analyses for the direction of any resulting bias in 



achievement are not entirely clear, an attempt 
was made to minimize any bias by incorporating 
the four variables in question into the adjustment 
for school nonresponse that is a component of 
the sampling weights. In the judgment of the 
international Technical Advisory Group this was 
sufficient to ensure that any remaining bias was 
likely to be minimal and hence that the data for 
the United States were included in the 
international database. A similar judgment was 
applied to the analyses conducted by the United 
Kingdom, but not for the Netherlands. 

Schools were contacted again approximately 1 
week before the assessment to select the student 
sample and arrange for assessment space in the 
school. Assessments were conducted in the United 
States in the spring of 2000 by trained test 
administration field staff that visited each of the 
participating schools and administered both the 
assessments and the questionnaires. 

Table Al.l provides summary information on the 
samples of all countries. A more detailed 
presentation can be found in the OECD’s 
forthcoming PISA 2000 technical report. 



Grade Distributions 

The students in PISA are selected on the basis 
that they are 15 years old and, as a result, are 
spread across several grades. Grade distributions 
for 15-year-olds vary from country to country as 
a function of policies about age of entry to school 
or other educational policies. The proportions of 
students at each grade level in each of the 
participating nations are shown in table A1.2. 

Quality Assurance 

PISA 2000 emphasized the use of standardized 
procedures in all countries. The PISA 
Consortium provided comprehensive manuals to 
explain the survey’s implementation, including 
precise instructions for the work of school co- 
coordinators and scripts for test administrators 
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for use in testing sessions. The quality and 
linguistic equivalence of instruments was 
ensured by providing countries with source 
versions of the assessment instruments in two 
languages (English and French) and 
recommending countries prepare and 
consolidate independent translations from both 
source versions, providing precise translation 
guidelines that included a description on the 
features each item was measuring and statistical 
analysis from the field trial. In cases where one 
source language was used, independent 
translations were required and discrepancies 
reconciled. The PISA Consortium verified the 
national translation and adaptation of all 
instrumentation. Additionally, members of the 
PISA Consortium visited all national centers to 
review data collection procedures, and members 
of the PISA Consortium visited a randomly 
selected subsample of 25 percent of the 
educational institutions. For a detailed 
description of the quality assurance procedures, 
see the OECD’s forthcoming technical report on 
PISA 2000. 

Test Development 

The development of the PISA 2000 assessment 
instruments was an interactive process among the 
PISA Consortium, various expert committees, and 
OECD governments. The intention was to reflect 
the national, cultural, and linguistic variety 
among OECD countries. The assessments 
included material selected from among items 
submitted by participating countries as well as 
items that were developed by the Consortium’s 
test developers. Each item, or question, was rated 
by each country on potential cultural, gender, or 
other types of bias. A small prepilot was 
v conducted in a limited number of countries prior 
to a field trial, which was conducted in all 
countries in 1999. Approximately 120 units 
(passages or pictures with related questions) were 
developed for the field trial, including more than 
800 reading items. The field trial included 69 
units with 342 items and the main study included 



37 units with 141 items. The mathematics 
instrument included 32 items; the science 
instrument included 35 items. 

The assessment instruments included curricular 
and noncurricular components following the 
framework specifications defined by subject 
matter experts (OECD 1999). One of the 
characteristics of the PISA 2000 instruments was 
the large amount of items requiring students to 
construct their own response. In reading, 45 
percent of items required an open-constructed 
response while this item type accounted for 35 
percent of the mathematics and science tests. Five 
item types were used in the PISA instruments: 
multiple choice, complex multiple choice, closed- 
constructed response, short response, and open- 
constructed response. 

The assessments were designed to yield group- 
level information in a broad range of content 
while meeting the limitation of 120 minutes of 
testing time per student. To achieve this goal, an 
unbalanced rotation design permitted an overall 
assessment of 270 minutes of reading, 60 minutes 
of mathematics, and 60 minutes of science. The 
assessment in each domain was divided into 
clusters, organized into nine booklets. There were 
nine 30-minute reading clusters, four 15-minute 
clusters of mathematics, and four 15-minute 
clusters of science. In PISA 2000, every student 
answered reading items; over half the students 
answered items on science and mathematics. 

This assessment design provides several features. 
First, the reading material was presented in a 
balanced way in order to avoid position effects 
and to ensure that each item had equal weight in 
the assessment. Second, seven of the nine 
booklets began with reading, and all booklets 
contained at least 60 minutes of reading. Five 
booklets also contained items for science, and 
five contained items for mathematics. Third, 
PISA 2000 included a link between PISA and 
LALS (the International Adult Literacy Study) 
through two reading blocks containing only 
IALS items, which were presented in six of the 



Outcomes of Learning 

PISA 2000 





:.gi--Notes 



mdix 



i 









nine booklets. Finally, this design ensures that a 
representative sample of students responded to 
each block of items. The OECD will publish 
further information on the PISA 2000 assessment 
design in a forthcoming technical report. 



Scoring 

PISA’s assessment of reading included 270 
minutes of testing time, of which 45 percent was 
devoted to items requiring open-ended 
responses. The mathematics and science tests 
included 60 minutes of testing time, of which 35 
percent was assessed through open-ended items. 
The process of scoring these items was an 
important step in ensuring the quality and 
comparability of the PISA data. 

Detailed guidelines were developed for the 
scoring guides themselves, training materials to 
recruit scorers, and workshop materials used for 
the training of national scorers. Prior to the 
national training, the PISA Consortium organized 
training sessions to present the material and train 
the scoring coordinators from the participating 
countries, who trained the national scorers. 

For each test item, the scoring guide described 
the intent of the question and how to code the 
students’ responses to each item. This description 
included the credit labels — full credit, partial 
credit, or no credit — attached to the possible 
categories of response. Also included was a 
system of double-digit coding for the 
mathematics and science items where the first 
digit represented the score, and the second digit 
represented different strategies or approaches 
that students used to solve the problem. The 
second digit generated national profiles of 
student strategies and misconceptions. In 
addition, the scoring guides included real 
examples of students’ responses accompanied by 
a rationale for their classification for purposes of 
clarity and illustration. 

To examine the consistency of this marking 
process in more detail within each country and to 



estimate the magnitude of the variance 
components associated with the use of markers, 
the PISA Consortium conducted an interscorer 
reliability study on a subsample of assessment 
booklets. Homogeneity analysis was applied to 
the national sets of multiple scoring and 
compared with the results of the field trial. A full 
description of this process and the results can be 
found in a technical report on PISA 2000 to be 
published by the OECD. 



Weighting 

Students included in the final PISA sample for a 
given country are not all equally representative 
of the full student population, even though 
random sampling of schools and students is used 
to select the sample. The use of sampling 
weights is necessary for the computation of 
statistically sound, nationally representative 
estimators. Survey weights help adjust for 
intentional over- or under-sampling of certain 
sectors of the population, school or student 
nonresponse, or errors in estimating size of a 
school at the time of sampling. 

For example, the United States over-sampled for 
minorities in public schools with 15 percent or 
more minority students in order to obtain 
enough data on these students to report 
accurately on them. Sampling weights were 
applied to the data to adjust for this over- 
sampling in order to ensure that the U.S. student 
sample represents the overall 15-year-old student 
population. The weight assigned to a student’s 
responses is the inverse of the probability that 
the student would be selected for the sample. 
When responses are weighted, none are 
discarded, and each contributes to the results for 
the total number of students represented by the 
individual student assessed. Weighting also 
adjusts for various situations, such as school and 
student nonresponse, because data cannot be 
assumed to be randomly missing. The 
internationally defined weighting specifications 
for PISA require that each assessed student’s 
sampling weight be the product of the inverse of 
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the school’s probability of selection, an 
adjustment for school-level nonresponse, the 
inverse of the student’s probability of selection, 
and an adjustment for student-level nonresponse. 
In addition, in the United States, two grade 
nonresponse factors were needed, one for grade 
9 and one for grade 10. All PISA analyses are 
conducted using these sampling weights. 

The procedures being used to derive the survey 
weights for PISA are in accordance with 
standards of best practice for the analysis of 
complex survey data. They correspond to 
procedures that are used to analyze survey data 
by the world’s major statistical agencies, as well 
as conforming to Westat’s own current best 
methods. These are also the procedures that 
have been used in previous international studies 
of educational achievement including TIMSS 
and TIMSS-R. 

Scaling and Plausible Values 

PISA used Item Response Theory (IRT) methods 
to produce scale scores that summarized the 
achievement results. PISA 2000 utilized a mixed 
coefficients multinomial logit IRT model to 
produce score scales that summarized the 
achievement results. This model is similar in 
principle to the more familiar two parameter IRT 
model. With this method, the performance of a 
sample of students in a subject area or subarea 
can be summarized on a single scale or a series of 
scales, even when different students are 
administered different items. Because of the 
reporting requirements for PISA and because of 
the large number of background variables 
associated with the assessment, a large number of 
analyses had to be conducted. The procedures 
PISA used for the analyses were developed to 
produce accurate results for groups of students 
while limiting the testing burden on individual 
students. Furthermore, these procedures provided 
data that could be readily used in secondary 
analyses. IRT scaling provides estimates of item 
parameters (e.g., difficulty, discrimination) that 
define the relationship between the item and the 



underlying variable measured by the test. 
Parameters of the IRT model are estimated for 
each test question, with an overall scale being 
established as well as scales for each predefined 
content area specified in the assessment 
framework. For example, PISA 2000 had four 
scales describing reading (a combined score and 
subscale scores in three domains) and one each 
for mathematics and science. 

Plausible Values 

During the scaling phase, plausible values were 
used to characterize scale scores for students 
participating in the assessment. To keep student 
burden to a minimum, PISA administered few 
assessment items to each student — too few to 
produce accurate content-related scale scores for 
each student. To account for this, PISA generated 
five possible scale scores for each student that 
represented selections from the distribution of 
scale scores of students with similar backgrounds 
who answered the assessment items the same 
way. The plausible-values technology is one way 
to ensure that the estimates of the average 
performance of student populations and the 
estimates of variability in those estimates are 
more accurate than those determined through 
traditional procedures, which estimate a single 
score for each student. During the construction of 
plausible values, careful quality control steps 
ensured that the subpopulation estimates based 
on these plausible values were accurate. 

It is important to recognize that plausible values 
are not test scores and should not be treated as 
such. Plausible values are random numbers that 
are drawn from the distribution of scores that 
could be reasonably assigned to each individual. 
As such, the plausible values contain random 
error variance components and are not optimal as 
scores for individuals. The PISA student file 
contains 30 plausible values, five for each of the 
five PISA 2000 cognitive scales (three reading 
subscales, one mathematics, and one science 
scale) and five for the combined reading scale. If 
an analysis is to be undertaken with one of these 
five cognitive scales, then (ideally) the analysis 
should be undertaken five times, once with each 



4 ± 



73 



ERIC 




Outcomes of Learning 

2 000 



echnicarNotes 



mdix i 



c 

H 

fl 

P 

T 

E 

R 

KD 

KD 

KD 

KD 

KD 

hO 

KD 

KD 

KD 




of the five relevant plausible values variables. 

The results of these five analyses are averaged and 
then significance tests that adjust for variation 
between the five sets of results are computed. 

PISA uses the plausible-values methodology to 
represent what the true performance of an 
individual might have been, had it been 
observed, using a small number of random 
draws from an empirically derived distribution of 
score values based on the student’s observed 
responses to assessment items and on 
background variables. Each random draw from 
the distribution is considered a representative 
value from the distribution of potential scale 
scores for all students in the sample who have 
similar characteristics and identical patterns of 
item responses. The draws from the distribution 
are different from one another to quantify the 
degree of precision (the width of the spread) in 
the underlying distribution of possible scale 
scores that could have caused the observed 
performances. The PISA plausible values function 
like point estimates of scale scores for many 
purposes, but they are unlike true point estimates 
in several respects. They differ from one another 
for any particular student, and the amount of 
difference quantifies the spread in the underlying 
distribution of possible scale scores for that 
student. Because of the plausible-values 
approach, secondary researchers can use the 
PISA data to carry out a wide range of analyses. 



Statistical Procedures 
T ests of significance 

Comparisons made in the text of this report have 
been tested for statistical significance. For 
example, in the commonly made comparison of 
country averages against the average of the 
United States, tests of statistical significance were 
used to establish whether or not the observed 
differences from the U.S. average were 
statistically significant. 

In almost all instances the tests used were 
standard Mests. These fell into two categories 



according to the nature of the comparison being 
made. In simple comparisons of country averages 
against the U.S. average or against the OECD 
average, the following formula was used to 
compute the t statistic: 

t - Est t - Est 2 / SQRTbse ^ 2 + (se 2 )] 2 

Est 1 and Est 2 are the estimates being compared 
(e.g., average of country A and the U.S. 
average) and se x and se 2 are the corresponding 
standard errors of these averages. 

In several places, between-country comparisons of 
group differences within countries were made. 
Comparisons of gender differences in other PISA 
countries against gender differences in the United 
States are an example. In these instances the 
following formula was used: 

t = (Est n - Est 21 ) - (Est 12 - Est 22 ) / SQRT[(se n 2 
+ se 21 2 ) + (se 12 2 + se 22 2 )] 

Est n and Est 21 are the estimates being 
compared within country A (e.g., female 
reading average and male reading average), 

Est 12 and Est 22 are the corresponding estimates 
for the United States, and se n , se 21 , se 12 , and 
se 22 are their corresponding standard errors. 

Since the International Socio-Economic Index 
(ISEI) is a continuous measure with no obvious 
cut-points that would allow the identification of 
socioeconomic groups, the linkage of 
socioeconomic status to average literacy scores is 
shown as a relationship rather than a difference 
between group averages. The measure of 
relationship in this case is a regression 
coefficient. For reading, mathematics, and 
science literacy measures, simple bivariate 
regressions were estimated within each country. 
The five plausible values available for each 
literacy measure were treated as the dependent 
variable and the ISEI index as the independent 
variable. These analyses were undertaken within 
Wesvar in order to obtain the correct standard 
errors for these statistics. Regression coefficients 
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for other PISA countries could then be tested 
against the U.S. coefficient. 

To guard against errors of inference based on 
multiple comparisons, as in the case of comparing 
all countries to the United States, the Bonferroni 
adjustment procedure was used. This procedure 
increases the critical value of t as the number of 
comparisons increases. For example, when 
making a simple pairwise comparison, the t value 
must be 1.96 or greater to achieve statistical 
significance at the .05 level of confidence. When 
making comparisons of all countries against the 
United States, the number of comparisons is 30 
and the critical value required for statistical 
significance is 3.14 at the .05 level. 

Standard errors 

The estimation of the standard errors that are 
required in order to undertake the tests of 
significance is complicated by the complex 
sample and assessment designs which both 
generate error variance. Together they mandate a 
set of statistically complicated procedures in 
order to estimate the correct standard errors. As 
a consequence, the estimated standard errors 
contain a sampling variance component 
estimated by Balanced Repeated Replication 
(BRR) — the Fay method of BRR; and, where the 
assessments are concerned, an additional 
imputation variance component arising from the 
assessment design. Details on the procedures 
used can be found in the WesVar 4.0 User's 
Guide (Westat 2000). 

Literacy Levels 

While the basic form of measurement in PISA 
describes student literacy in each country in terms 
of a range of scale scores, PISA also treats 
proficiency in reading literacy in terms of five 
levels, each representing tasks of increasing 
complexity. As a result, the literacy findings are 
reported in terms of percentages of the 
population proficient at handling tasks of 
different levels of difficulty. 



Each of the four reading literacy scales — the 
combined score and the three subscale scores — is 
divided into five levels based on the type of 
knowledge and skills students need to 
demonstrate at a particular level. Cut scores for 
the levels are as follows: below level 1: a score 
equal to or less than 334.75; level 1: a score 
greater than 334.75 and equal to or below 
407.47; level 2: a score greater than 407.47 and 
equal to or below 480.18; level 3: a score 
greater than 480.18 and equal to or below 
552.89; level 4: a score greater than 552.89 and 
equal to or below 625.61; and level 5: a score 
greater than 625.61. 

All students within a level are expected to 
answer at least half of the items from that level 
correctly. Students at the bottom of a level have a 
62 percent chance of success on the easiest items 
from that level and a 42 percent chance of 
success on the hardest items from that level. 
Students at the top of a level are able to provide 
the correct answers to about 70 percent of all 
items from that level, have a 62 percent chance 
of success on the hardest items from that level, 
and have a 78 percent chance of success on the 
easiest items from that level. Students just below 
the top of a level would score less than 50 
percent on an assessment of the next higher level. 
Students at a particular level not only 
demonstrate the knowledge and skills associated 
with that level but also the proficiencies defined 
by lower levels. Thus, all students proficient at 
level 3 are also proficient at levels 1 and 2. All 
students within a level are expected to answer at 
least half of the items in that level correctly. 
Patterns of responses for students below level 1 
suggest they are unable to answer at least half of 
the items in level 1 correctly. 
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Table A2. 1 . — Descriptions of international assessments of reading 







PISA 


PIRLS 


ALL 


Age 


1 5-year-olds 


9-year-olds 


16 through 65-year-olds 


Periodicity 


Every 3 years 

2000, 2003, 2006, 2009... 


Every 4 years 
2001, 2005, 2009... 


Follow-up to International Adult 
Literacy Study (IALS) 

1994, 2002... 


Domains 


Reading literacy, mathematics 
literacy, and science literacy 


Reading literacy 


Document literacy, prose literacy, 
numeracy, and analytic reasoning 


Definition 


Understanding, using, and 
reflecting on written texts in order 
to achieve one's goals, to develop 
knowledge and potential, and to 
participate in society. 


Understanding and using those 
written language forms required 
by society and/or valued by the 
individual. Young readers can 
construct meaning from a variety 
of texts. They read to learn, to 
participate in a community of 
readers, and for enjoyment. 


Using printed and written 
information to function in 
society to achieve one's goals 
and to develop one's 
knowledge and potential. 


Framework 

components 


Types of text: (1) continuous, e.g., 
narrative and (2) non-continuous, 
e.g., map 


Processes: ( 1 ) focus on and 
retrieve explicitly stated 
information; (2) make 
straightforward inferences; (3) 
interpret and integrate ideas and 
information; and (4) examine and 
evaluate context, language, and 
text elements 


Prose literacy: (1) locating 
information; (2) integrating 
information; and 
(3) generating information 


Situations: (1) Reading for public 
use; (2) reading for private use; 
(3) reading for work; and (4) 
reading for education 


Document literacy: (1) locating 
information; (2) cycling 
information; (3) integrating 
information; and 
(4) generating information 


Purposes of reading: 

(1) reading for literary experience 
and (2) reading to acquire and 
use information 


Processes: (1) retrieving 
information; (2) broad 
understanding; (3) developing 
an interpretation; (4) reflecting 
on content; and (5) reflecting 
on form 


Contexts and contents: ( 1 ) home 
and family; (2) health and safety; 
(3) community and citizenship; (4) 
consumer electronics; (5) work; 
and (6) leisure and recreation 



Reporting 


Overall and processes of reading 


Overall and purposes of reading 


Prose and document literacy 


scales 










NOTE: PISA is the Program for International Student Assessment. PIRLS is the Progress in International Reading Literacy Study. ALL is 
the Adult Literacy and Lifeskills survey. 

SOURCE: Adult Literacy and Lifeskills survey (2001b). Prose and Document Literacy Framework. Available: 
ftp^/etsisl .ets.org/pub/corp/Prose-final.pdf; Campbell, J.R., Kelly, D.L., Mullis, I.V.S., Martin, M.O., and Sainsbury, M. (2001). 
Framework and Specifications for PIRLS Assessment zooi. Chestnut Hill, MA: Boston College; Organization for Economic 
Cooperation and Development (1999). Measuring Student Knowledge and Skills: A New Framework for Assessment. Paris: Author. 
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Table A2.2. — Descriptions of international assessments of mathematics 





PISA 


TIMSS-R 


ALL 


Age 


15-year-alds 


1 3-year-alds 


16 through 65-year-alds 


Periodicity 


Every 3 years 

2000, 2003, 2006, 2009... 


Every 4 years 
1995, 1999, 2003... 


Fallow- up to International Adult 
Literacy Study (IALS) 

1994, 2002... 


Domains 


Reading literacy, mathematics 
literacy, and science literacy 


Mathematics and science 


Document literacy, prase literacy, 
numeracy, and analytic reasoning 


Definition 


An individual's capacity to iden- 
tify and understand the rale that 
mathematics plays in the world, to 
make well-founded judgments and 
to engage in mathematics, in ways 
that meet the needs af that 
individual's life as a constructive, 
concerned and reflective citizen. 


Curricular mathematics as defined 
by content areas (see framework 
components) agreed upon by 
participating countries. 


The knowledge and skills 
required to effectively manage 
the mathematical demands of 
diverse situations. 


Framework 

components 


Major topics: (1 ) quantity; (2) 
space and shape; (3) change and 
relationships; and (4) uncertainty 


Content areas: (1 ) fractions and 
number sense; (2) data 
representation, analysis, and 
probability; (3) algebra; (4) 
measurement; and (5) geometry 


Facets: numerate behavior 
involves managing a situation ar 
salving a problem in a real 
context by responding to 
mathematical information that is 
represented in a range of ways 
and requires activation af a range 
of enabling knowledge, 
behaviors, and processes. 


Competency classes/processes: 

(1 ) reproduction and routine 
procedures; (2) connections and 
integration far standard problem 
salving; and (3) reasoning, 
argumentation, insight and 
generalization far original 
problem salving 


Performance expectations: 

(1) knowing; (2) using routine 
procedures; (3) investigation and 
salving problems; (4) mathema- 
tical reasoning; and 
(5) communicating 


Situations: (1) personal; (2) 
educational; (3) occupational; (4) 
public; and (5) scientific 


Reporting 

scales 


Overall and by sub- scales to be 
determined (2003) 


Overall and by content area 


Numeracy 



NOTE: PISA is the Program for International Student Assessment. TIMSS-R is the Third International Mathematics and Science Study- 
Repeat. ALL is the Adult Literacy and Lifeskills survey. 

SOURCE: Adult Literacy and Lifeskills survey (2001a). Numeracy Framework. Available: 

ftp://etsisLets.org/pub/corp/Numeracy%20Framework.pdf; Organization for Economic Cooperation and Development (1999). 
Measuring Student Knowledge and Skills: A New Framework for Assessment . Paris: Author; Robitaille, D.F., Schmidt, W.H., Raizen, 
S., McKnight, C., Britton, E., and Nicol, C. (1993). Curriculum Frameworks for Mathematics and Science . TIMSS monograph no. 1. 
Vancouver, BC: Pacific Education Press. 
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Table A2.3. — Descriptions of international assessments of science 



C 

H 

fl 

P 

T 

E 

R 



<D 



<D 

<D 

<D 

<D 

<nT) 



r a 





PISA 


TIMSS-R 


Age 


1 5-year-olds 


1 3-year-olds 


Periodicity 


Every 3 years 

2000, 2003, 2006, 2009... 


Every 4 years 
1995, 1999, 2003... 


Domains 


Reading literacy, mathematics 
literacy, and science literacy 


Mathematics and science 


Definition 


The capacity to use scientific 
knowledge, to identify questions, 
and to draw evidence-based 
conclusions in order to understand 
and help make decisions about 
the natural world and the changes 
made to it through human activity. 


Curricular science as defined by 
content areas (see framework 
components) agreed upon by 
participating countries. 


Framework 

components 


Scientific applications: (1) science 
in life and health; (2) science in 
Earth and environment; and (3) 
science in technology 


Content areas: (1) earth science; 
(2) life science; (3) physics; (4) 
chemistry; (5) environment and 
resource issues; and (6) scientific 
inquiry and the nature of science 










Scientific processes: (1) 
recognizing scientifically 
investigable questions; (2) 
identifying evidence needed in a 
scientific investigation; (3) 
drawing or evaluating 
conclusions; (4) communicating 
valid conclusions; and (5) 
demonstrating comprehension of 
scientific concepts 


Performance expectations: 

(1) understanding simple 
information; (2) understanding 
complex information; (3) 
theorizing, analyzing, and solving 
problems; (4) using tools, routine 
procedures, and science 
processes; and (5) investigating 
the natural world 



Situations: (1) personal; (2) 
public; (3) global; and (4) 
historical relevance 



Reporting 


Overall and by sub-scales to be 


Overall and by content area 


scales 


determined (2006) 





V J 



NOTE: PISA is the Program for International Student Assessment. TIMSS-R is the Third International Mathematics and Science Study- 
Repeat. The framework for the Program for International Student Assessment (PISA) does include a list of 13 concepts to be assessed by 
PISA. However, in PISA 2000 these concepts are assessed through the scientific applications listed above. 

SOURCE: Organization for Economic Cooperation and Development (1999). Measuring Student Knowledge and Skills : A New 
Framework for Assessment. Paris: Author; Martin, M.O., Mullis, I.V.S., Gonzalez, E J., Gregory, K.D., Smith, T.A., Chrostowski, S.J., 
Garden, R.A., and O’Connor; K.M. (2000). TIMSS 19 99 International Science Report: Findings from IEAs Repeat of the Third 
International Mathematics and Science Study at the Eighth Grade. Chestnut Hill, MA: Boston College. 
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Table A3.6. — Percentage distribution and standard errors of 1 5-year-olds by retrieving information level, by country: 2000 
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Table A3. 8. — Percentage distribution and standard errors of 1 5-year-olds by reflecting on texts level, by country: 2000 
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Table A3.1 2. — Percentage of 1 5-year-olds reaching PISA international benchmarks in mathematics literacy with standard errors, by country: 2000 
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Table A3. 14. — Combined reading literacy averages by gender with standard errors, by country: 2000 
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Table A3.1 6. — Averages on the interpreting texts reading literacy subscale by gender with standard errors, by country: 2000 
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Table A3. 24. — Combined reading literacy averages by parents 7 national origin with standard errors, by country: 
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Table A3. 30. — Reading, mathematics and science literacy averages of U.S. 15-year-olds by race/ ethnicity with standard errors: 2( 
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RELERSED ITEMS 



Selected Sample Items 

These PISA sample items illustrate a range of questions across 
the three PISA domains — reading, mathematics and science 
literacy. Reading literacy proficiency is described by 5 levels, 
which represent increasing ability to solve problems in a real- 
life context as one moves from level 1 to 5. PISA will develop 
similar levels for mathematics and science literacy when each 
is the major content area (mathematics in 2003 and science 
in 2006). In addition, for reading literacy information is 
provided on specific reading processes or aspects such as 
retrieving information, interpreting texts, and reflecting on 
texts. Each sample item displayed for reading literacy 
describes its level and which specific reading process or 
aspect it assesses. 
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FEEL GOOD IN YOUR RUNNERS 
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For 14 years the Sports Medicine Center of Lyon (France) has been studying the 
injuries of young sports players and sports professionals. The study has established 
that the best course is prevention . . . and good shoes. 
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Knocks, falls, wear and 
tear... 

Eighteen percent of sports 
players aged 8 to 12 already 
have heel injuries. The 
cartilage of a soccer player’s 
ankle does not respond well 
to shocks, and 25% of 
professionals have 
discovered for themselves 
that it is an especially weak 
point. The cartilage of the 
delicate knee joint can also 
be irreparably damaged and 
if care is not taken right from 
childhood (10-12 years of 
age), this can cause 
premature osteoarthritis. The 
hip does not escape damage 
either and, particularly when 
tired, players run the risk of 
fractures as a result of falls 
or collisions. 

According to the study, 
soccer players who have 
been playing for more than 
ten years have bony 



outgrowths either on the tibia 
or on the heel. This is what 
is known as “soccer player’s 
foot”, a deformity caused by 
shoes with soles and ankle 
parts that are too flexible. 

Protect, support, stabilize, 
absorb 

If a shoe is too rigid, it 
restricts movement. If it is 
too flexible, it increases the 
risk of injuries and sprains. 

A good sports shoe should 
meet four criteria: 

Firstly, it must provide 
exterior protection: resisting 
knocks from the ball or 
another player, coping with 
unevenness in the ground, 
and keeping the foot warm 
and dry even when it is 
freezing cold and raining. 

It must support the foot , and 
in particular the ankle joint, to 
avoid sprains, swelling and 



other problems, which may 
even affect the knee. 

It must also provide players 
with good stability so that 
they do not slip on a wet 
ground or skid on a surface 
that is too dry. 

Finally, it must absorb 
shocks, especially those 
suffered by volleyball and 
basketball players who are 
constantly jumping. 

Dry feet 

To avoid minor but painful 
conditions such as blisters or 
even splits or athlete’s foot 
(fungal infections), the shoe 
must allow evaporation of 
perspiration and must prevent 
outside dampness from 
getting in. The ideal material 
for this is leather, which can 
be waterproofed to prevent 
the shoe from getting soaked 
the first time it rains. 
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Use the article on the opposite page to answer the questions below. 



RELEASED QUESTION 26: RUNNERS 

Aspect: Interpreting texts 
Level: 1 

Text format: Continuous 
Situation: Educational 



What does the author intend to show in this text? 

A That the quality of many sports shoes has greatly improved. 

B That it is best not to play soccer if you are under 12 years of age. 

C That young people are suffering more and more injuries due to their poor 
physical condition. 

D That it is very important for young sports players to wear good sports shoes. 



The correct answer is D. 



United States percent full credit (s.e.) 78 (2.2) 

OECD percent full credit (s.e.) 87 (0.2) 



NOTE: s.e. means standard error. 



RELEASED QUESTION 28: RUNNERS 

Aspect: Retrieving information 
Level: 1 

Text format: Continuous 
Situation: Educational 



One part of the article says, “A good sports shoe should meet four criteria.” 
What are these criteria? 



To receive full credit, students must refer to the four criteria in italics in 
the text. 
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United States percent full credit (s.e.) 84 (1 .5) 

OECD percent full credit (s.e.) 80 (0.2) 



NOTE: s.e. means standard error. 
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RELEASED QUESTION 29: RUNNERS 

C Aspect: Reflecting on texts 

pi Level: 1 

p Text format: Continuous 

p Situation: Educational 

T 

E 

r Look at this sentence from near the end of the article. It is presented here in two parts: 




“To avoid minor but painful conditions such as blisters or even 
splits or athlete's foot (fungal infections),...” 



(first part) 
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“...the shoe must allow evaporation of perspiration and must (second part) 

prevent outside dampness from getting in.” 
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What is the relationship between the first and second parts of the sentence? 

The second part 

A contradicts the first part. 

B repeats the first part. 

C illustrates the problem described in the first part. 

D gives the solution to the problem described in the first part. 
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The correct answer is D. 



United States percent full credit (s.e.) 78 (1 .5) 

OECD percent full credit (s.e.) 79 (0.2) 



NOTE: s.e. means standard error. 
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ACOL VOLUNTARY FLU IMMUNIZATION PROGRAM 

As you are no doubt aware the flu can strike rapidly and extensively during winter. It 
can leave its victims ill for weeks. 

The best way to fight the virus is to have a fit and healthy body. Daily exercise and a 
diet including plenty of fruits and vegetables are highly recommended to assist the 
immune system to fight this invading virus. 

— <==N 

ACOL has decided to offer staff the opportunity to be immunized against the flu as an 
additional way to prevent this insidious virus from spreading amongst us. ACOL has 
arranged for a nurse to administer the immunizations at ACOL, during a half-day 
session in work hours in the week of October 17. This program is free and available to 
all members of staff. 

Participation is voluntary. Staff exercising the option will be asked to sign a consent 
form indicating that they do not have any allergies, and that they understand they may 
experience minor side effects. 

Medical advice indicates that the immunization does not produce influenza. However, it 
may cause some side effects such as fatigue, mild fever and tenderness of the arm. 



WHO SHOULD BE IMMUNIZED? 

Anyone interested in being protected against the virus. 

This immunization is especially recommended for people over the age of 65. But 
regardless of age, ANYONE who has a chronic debilitating disease, especially cardiac, 
pulmonary, bronchial or diabetic conditions. 

In an office environment ALL staff are at risk of catching the flu. 

WHO SHOULD NOT BE IMMUNIZED? 

Individuals hypersensitive to eggs, people suffering from an acute feverish illness and 
pregnant women. 

Check with your doctor if you are taking any medication or have had a previous 
reaction to a flu injection. 



If you would like to be immunized in the week of October 17 please advise the 
personnel officer, Anne Washington, by Friday October 7. The date and time will be set 
according to the availability of the nurse, the number of participants and the time 
convenient for most staff. If you would like to be immunized for this winter but cannot 
attend at the arranged time please let Anne know. An alternative session may be 
arranged if there are sufficient numbers. 

For further information please contact Anne on ext. 5577. 
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Anne Washington, the personnel officer at a company called ACOL, prepared the 
information sheet on the previous page for ACOL staff. Refer to the information sheet 
to answer the questions which follow. 

C ^ 

H RELEASED QUESTION 6: FLU 

ft 

Aspect: Retrieving information 
L. Level: 2 

Text format: Continuous 
^ Situation: Occupational 
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Which one of the following describes a feature of the ACOL flu immunization program? 
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A Daily exercise classes will be run during the winter. 
B Immunizations will be given during working hours. 
C A small bonus will be offered to participants. 

D A doctor will give the shots. 
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The correct answer is B. 



United States percent full credit (s.e.) 71 (2.9) 

OECD percent full credit (s.e.) 73 (0.3) 



NOTE: s.e. means standard error. 
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RELEASED QUESTION 7: FLU 

Aspect: Reflecting on texts 
Level: 3 (full credit) 3 (partial credit) 

Text format ; Continuous 
Situation: Occupational 



We can talk about the content of a piece of writing (what it says). 

We can talk about its style (the way it is presented). 

Anne wanted the style of this information sheet to be friendly and encouraging. 
Do you think she succeeded? 

Explain your answer by referring in detail to the layout, style of writing, pictures 
or other graphics. 



To receive full credit, responses must refer accurately to the text and 
relate style to purpose, consistent with "friendly and encouraging." The 
answer must do at least one of the following: 1 ) refer to one of the 
features in detail and/or 2) use evaluative terms other than "friendly 
and encouraging." Opinion about whether Anne succeeded may be 
stated or implied. 

Responses that refer accurately to the text and relate purpose to 
information and content (rather than style), consistent with "friendly and 
encouraging," receive partial credit. Opinion about whether Anne 
succeeded may be stated or implied. 



United States percent full credit (s.e.) 37 (2.7) 

OECD percent full credit (s.e.) 48 (0.4) 

United States percent partial credit (s.e.) 16 (1 .8) 

OECD percent partial credit (s.e.) 17 (0.3) 



NOTE: s.e. means standard error. 
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RELEASED QUESTION 8: FLU 

Aspect: Interpreting texts 
Level: 3 

Text format: Continuous 
Situation: Occupational 

This information sheet suggests that if you want to protect yourself against the flu virus, 
a flu shot is 
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A more effective than exercise and a healthy diet, but more risky. 

B a good idea, but not a substitute for exercise and a healthy diet. 

C as effective as exercise and a healthy diet, and less troublesome. 

D not worth considering if you have plenty of exercise and a healthy diet. 
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The correct answer is B. 



United States percent full credit (s.e.) 70 (2.4) 

OECD percent full credit (s.e.) 56 (0.3) 



NOTE: s.e. means standard error. 
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RELEASED QUESTION 9: FLU 

Aspect: Reflecting on texts 
Level: 4 

Text format: Continuous 
Situation: Occupational 



Part of the information sheet says: 

WHO SHOULD BE IMMUNIZED? 

Anyone interested in being protected against the virus. 

After Anne had circulated the information sheet, a colleague told her that she should 
have left out the words “Anyone interested in being protected against the virus” 
because they were misleading. 

Do you agree that these words are misleading and should have been left out? 
Explain your answer. 



To receive full credit, responses should evaluate the section of text in 
relation to the term "misleading" by indicating that there is a 
contradiction. This contradiction may or may not be explained and 
agreement or disagreement may be stated or implied. 

Full credit can also be gained by evaluating the section of text in 
relation to the term "misleading" by indicating that the statement may 
be an exaggeration. This exaggeration may or may not be explained 
and agreement or disagreement may be stated or implied. 



United States percent full credit (s.e.) 41 (2.8) 

OECD percent full credit (s.e.) 35 (0.3) 



NOTE: s.e. means standard error. 
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LABOR 

The tree diagram below shows the structure of a country’s labor force or “working-age 
population”. The total population of the country in 1995 was about 3.4 million. 



The Labor Force Structure year ended March 31, 1995 (000s) 1 




Notes 

1 . Numbers of people are given in thousands (000s). 

2. The working-age population is defined as people between the ages of 15 and 65. 

3. People “Not in labor force” are those not actively seeking work and/or not available for work. 
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Use the information about a country’s labor force on the opposite page to answer the 
questions below. 



RELEASED QUESTION 16: LABOR 

Aspect: Retrieving information 
Level: 4 (full credit) 3 (partial credit) 

Text format: Non-continuous 
Situation: Educational 



How many people of working age were not in the labor force? (Write the number of 
people, not the percentage.) 



To receive full credit, responses should indicate that the number in the 
tree diagram and the "000s" in the title/ footnote have been integrated 
to produce the number 949,000. Approximations between 949,000 
and 950,000, in numbers or words, were accepted. In addition, 
900,000 or one million (in words or numbers) were accepted if 
accompanied by a qualifier such as "almost" or "about." 

Correctly indicating the number in the tree diagram, but failing to 
integrate the "000s" in the title/footnote can achieve partial credit. In 
this situation, students would answer 949.9 in words or numbers. 



United States percent full credit (s.e.) 24 (2.1 ) 

OECD percent full credit (s.e.) 31 (0.4) 

United States percent partial credit (s.e.) 54 (2.3) 

OECD percent partial credit (s.e.) 41 (0.4) 



NOTE: s.e. means standard error. 
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RELEASED QUESTION 17: LABOR 

Aspect: Interpreting texts 

Level: 5 (full credit) 2 (partial credit) 

Text format: Non-continuous 
Situation: Educational 

In which part of the tree diagram, if any, would each of the people listed in the table 
below be included? 

Show your answer by placing a cross in the correct box in the table. 

The first one has been done for you. 



KD 

KD 

KD 

KD 

KD 

KD 





"In labor 


"In labor 


"Not in 


Not 




force: 


force: 


labor 


included in 




employed" 


unemployed” 


force” 


any category 


A part-time waiter, aged 35 


IEI 


□ 


□ 


□ 


A business woman, aged 43, who works a 
sixty-hour week 


□ 


□ 


□ 


□ 


A full-time student, aged 21 


□ 


□ 


□ 


□ 


A man, aged 28, who recently sold his 
shop and is looking for work 


□ 


□ 


□ 


□ 


A woman, aged 55, who has never worked 
or wanted to work outside the home 


□ 


□ 


□ 


□ 


A grandmother, aged 80, who still works a 
few hours a day at the family’s store 


□ 


□ 


□ 


□ 




On the following page is a correctly answered diagram. 
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A part-time waiter, aged 35 

A business woman, aged 43, who works a 
sixty-hour week 

A full-time student, aged 21 

A man, aged 28, who recently sold his 
shop and is looking for work 

A woman, aged 55, who has never worked 
or wanted to work outside the home 

A grandmother, aged 80, who still works a 
few hours a day at the family’s store 



"In labor 


"In labor 


“Not in 


Not 


force: 


force: 


labor 


included in 


employed" 


unemployed" 


force" 


any category 




X 




□ 


□ 


□ 




X 




□ 


□ 


□ 



□ 


□ 




□ 


□ 




□ 


□ 


□ 


□ 


X 


□ 


□ 


□ 


□ 


X 



To receive full credit, students needed to correctly identify each person's 
labor classification. Students who correctly identified 3 or 4 of the 
labor classifications received partial credit. 



United States percent full credit (s.e.) 15(1.9) 

OECD percent full credit (s.e.) 14 (0.3) 

United States percent partial credit (s.e.) 53 (2.7) 

OECD percent partial credit (s.e.) 54 (0.3) 



NOTE: s.e. means standard error. 
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AMANDA AND THE DUCHESS 

Summary: Since Leocadia’s death, the Prince, who was in love with her, has been inconsolable. At a 
shop called R6seda Soeurs, the Duchess, who is the Prince's aunt, has met a young shop assistant, 
Amanda, who looks amazingly like Leocadia. The Duchess wants Amanda to help her set the Prince 
free from the memories which haunt him. 



A crossroads in the castle grounds, a circular bench 
around a small obelisk. . .evening is falling... 

AMANDA 

I still don't understand. What can I do for him, 
ma'am? I can't believe you could possibly have 
thought... And why me? I'm not particularly 
pretty. And even if someone were very 
pretty — who could suddenly come between 
him and his memories like that? 

THE DUCHESS 
No-one but you. 

AMANDA, sincerely surprised 

Me? 

THE DUCHESS 

The world is so foolish, my child. It sees only 
parades, gestures, badges of office.. .that must 
be why you have never been told. But my heart 
hasn't deceived me — I almost cried out at 
Reseda Soeurs the first time I saw you. To 
someone who knew more of her than just her 
public image, you are the living likeness of 
Leocadia. A silence. The evening birds have nozo 
taken over from the afternoon birds. The grounds are 
filled zoith shadozvs and tzvittering. 

AMANDA, very gently 
I really don't think I can, ma'am. I have 
nothing, I am nothing, and those lovers. . .that 
was my fancy, don't you see? She has got up. As 
if about to leave, she has picked up her small 
suitcase. 

THE DUCHESS, gently also, and very zvearily 
Of course, my dear. I apologize. 

She in turn gets up, zvith difficulty , like an old 
zvoman. A bicycle bell is heard in the evening air; she 
gives a start. 

Listen. . .it's him! Just show yourself to him, 
leaning against this little obelisk where he first 
met her. Let him see you, even if it's just this 
once, let him call out, take a sudden interest in 
this likeness, in this stratagem which I shall 
confess to him tomorrow and for which he will 
hate me — in anything but this dead girl who'll 
take him away from me one of these days. I'm 
sure. . .(She has taken her by the arm.) You will do 
that, won't you? I beg you most humbly, young 
lady. (She looks at her, beseechingly, and quickly 
adds:) And then, that way, you'll see him too. 
And. . .1 can feel that I'm blushing again from 



saying this to you — life is just too mad! That's 
the third time I've blushed in sixty years, and 
the second time in ten minutes — you'll see him; 
and if he could ever (why not him, since he's 
handsome and charming and he wouldn't be 
the first?) if he could ever have the good 
fortune, for himself and for me, to take your 
fancy for one moment. . .The bell again in the 
shadozvs, but very close nozo. 

AMANDA, in a zvhisper 
What should I say to him? 

THE DUCHESS, gripping her arm 

Simply say: "Excuse me. Sir, can you tell me the 

way to the sea?" She has hurried into the deeper 

shadozvs of the trees. Just in time. There is a pale 

blur. It is the Prince on his bicycle. He passes very 

close to the pale blur of Amanda by the obelisk. She 

murmurs. 

AMANDA 

Excuse me. Sir... 

He stops, dismounts from the bicycle, takes off his 
hat and looks at her. 

THE PRINCE 

Yes? 

AMANDA 

Can you tell me the way to the sea? 

THE PRINCE 

Take the second turning on your left. 

He bozvs, sadly and courteously, gets back on the 
bicycle and rides azvay. The bell is heard again in the 
distance. The Duchess comes out of the shadozvs, 
very much an old zvoman. 

AMANDA, gently, after a zvhile 
He didn't recognize me. . . 

THE DUCHESS 

It was dark. . .And then, who knows what face 
he gives her now, in his dreams? (She asks 
timidly:) The last train has gone, young lady. In 
any case, wouldn't you like to stay at the castle 
tonight? 

AMANDA, in a strange voice 
Yes, ma'am. It is completely dark. The tzvo of them 
can no longer be seen in the shadozvs, and only the 
zvind can be heard in the huge trees of the grounds. 

THE CURTAIN FALLS 
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RELEASED QUESTION 40: AMANDA AND THE DUCHESS 

Aspect: Retrieving information 
Level: 4 

Text format: Con tinuous 
Situation: Personal 



The director positions the actors on the stage. On a diagram, the director represents 
Amanda with the letter A and the Duchess with the letter D. 

Put an A and a D on the following diagram of the set to show approximately where 
Amanda and the Duchess are when the Prince arrives. 




To receive full credit, the students must correctly mark an "A" by the 
obelisk and a "D" behind or near the trees. 



United States percent full credit (s.e.) 38 (2.9) 

OECD percent full credit (s.e.) 47 (0.4) 



NOTE: s.e. means standard error. 
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CONTINENT AREA 



Below is a map of Antarctica. 



<•* 
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RELEASED QUESTION 4: CONTINENT AREA 

Competency class 2: Connections and integration for problem solving 
Overarching concept: Space and shape 
Situation: Personal 



Estimate the area of Antarctica using the map scale. 

Show your work and explain how you made your estimate. 

(You can draw over the map if it helps you with your estimation) 



To receive full credit, students needed to provide the correct answer, 
between 1 2,000,000 square kilometers and 1 8,000,000 square 
kilometers (units not required), and could show evidence of using a 
correct method such as drawing a square or circle to estimate the 
answer. Students who provided the correct answer but did not show 
any work also received full credit. 

Students who show evidence of using a correct method, but provided 
an incorrect answer, received partial credit. 



United States percent full credit (s.e.) 10 (1 .3) 

OECD percent full credit (s.e.) 20 (0.3) 

United States percent partial credit (s.e.) 38 (2.5) 

OECD percent partial credit (s.e.) 40 (0.4) 



NOTE: s.e. means standard error. 
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TRIANGLES 



RELEASED QUESTION 9: TRIANGLES 

Competency class 1: Reproduction , definitions and computations 
Overarching concept: Space and shape 
Situation: Scientific 



Circle the one figure below that fits the following description. 

Triangle PQR is a right triangle with right angle at R. The segment RQ is less 
than the segment PR. M is the midpoint of the segment PQ and N is the midpoint of 
the segment QR. S is a point inside the triangle. The segment MN is greater than the 
segment MS. 




The correct answer is D. 



United States percent full credit (s.e.) 46 (1 .9) 

OECD percent full credit (s.e.) 62 (0.3) 



NOTE: s.e. means standard error. 
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OZONE 

Read the following section of an article about the ozone layer . : 



The atmosphere is an ocean of air and a precious natural resource for sustaining life 
on the Earth. Unfortunately, human activities based on national/personal interests 
are causing harm to this common resource, notably by depleting the fragile ozone 
layer, which acts as a protective shield for life on the Earth. 

5 Ozone molecules consist of three oxygen atoms, as opposed to oxygen molecules 
which consist of two oxygen atoms. Ozone molecules are exceedingly rare: fewer 
than ten in every million molecules of air. However, for nearly a billion years, their 
presence in the atmosphere has played a vital role in safeguarding life on Earth. 
Depending on where it is located, ozone can either protect or harm life on Earth. 

10 The ozone in the troposphere (up to 10 kilometers above the Earth's surface) is 
"bad" ozone which can damage lung tissues and plants. But about 90 percent of 
ozone found in the stratosphere (between 10 and 40 kilometers above the Earth's 
surface) is "good" ozone which plays a beneficial role by absorbing dangerous 
ultraviolet (UV-B) radiation from the Sun. 

Without this beneficial ozone layer, humans would be more susceptible to certain 
15 diseases due to the increased incidence of ultra-violet rays from the Sun. In the last 
decades the amount of ozone has decreased. In 1974 it was hypothesized that 
chlorofluorocarbons (CFCs) could be a cause for this. Until 1987, scientific 
assessment of the cause-effect relationship was not convincing enough to implicate 
CFCs. However, in September 1987, diplomats from around the world met in 
20 Montreal (Canada) and agreed to set sharp limits to the use of CFCs. 
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RELEASED QUESTION 7: OZONE 

Process: Apply scientific knowledge in situation presented 
Area: Science in life and health (physiological change) 
Situation: Global 



Lines 14 and 15 state: “Without this beneficial ozone layer, humans would be more 
susceptible to certain diseases due to the increased incidence of ultra-violet rays 
from the Sun.” 

Name one of these specific diseases. 



To receive full credit, answers must refer to skin cancer or melanoma. 



United States percent full credit (s.e.) 63 (2.5) 

OECD percent full credit (s.e.) 63 (0.3) 



NOTE: s.e. means standard error. 
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RELEASED QUESTION 8: OZONE 

Process: Recognizing questions 

Area: Science in Earth and environment (Earth/Space) 

Situation: Global 



At the end of the text, an international meeting in Montreal is mentioned. At that 
meeting lots of questions in relation to the possible depletion of the ozone layer 
were discussed. Two of those questions are given in the table below. 

Can the questions listed below be answered by scientific research? 



Circle Yes or No for each. 



Question: 


Answerable by 

scientific 

research? 


Should the scientific uncertainties about the influence of 
CFCs on the ozone layer be a reason for governments 
to take no action? 


Yes / No 


What would the concentration of CFCs be in the atmosphere 
in the year 2002 if the release of CFCs into the atmosphere 
takes place at the same rate as it does now? 


Yes / No 



To receive full credit, students must answer "no" to the first question and 
"yes" to the second question. 



United States percent full credit (s.e.) 64 (2.1 ) 

OECD percent full credit (s.e.) 59 (0.3) 



NOTE: s.e. means standard error. 
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